How AI has learned human emotions from art

ArtEmis models how people describe their emotional responses to artworks. Its generated explanations are predictions learned from human examples, rather than evidence that the system experiences those feelings. For museums, the interesting question is how such descriptions might support interpretation while leaving room for visitors to respond differently.

Predicting how people respond to art

The study conducted by Stanford University’s Institute for Human-Centered AI trained their artificial intelligence, called ArtEmis, to predict descriptions of human emotional responses to artworks.

By drawing from a dataset of over 81,000 paintings and 440,000 human responses, the AI algorithm learned to predict how a human might respond to the artwork – and then to justify this prediction in words. The output is a prediction based on human descriptions, rather than evidence of felt empathy.

For Van Gogh’s Starry Night, the AI responded with ‘Awe. The blue and white colours of this painting make me feel like I am looking at a dream’.

When shown Dali’s The Persistence of Memory, ArtEmis replied with ‘Fear. It looks like an animal is laying dead on the ground’.

The generated descriptions can refer to different parts of a work and model the different responses people may have to the same image.

AI does not have a great reputation for its emotional intelligence. Emotion recognition technology is on the rise and has masses of potential, from marketeers tracking facial responses to law courts using AI to decipher the expressions of those standing trial. However, it’s also fraught with issues such as extremely troubling racial bias, simplification of psychological states, and dehumanization. This last has been termed ‘emojification’ by Dove Tail Labs in their Emotion Recognition System, which seeks to open up discourse around this technology (and the website has a brilliant – and worrying – AR camera to try yourself).

ArtEmis explores how AI can model the language people use to describe emotional responses.

And the creators can see a myriad of uses for it, from helping designers to test the effectiveness of their work, to enriching chatbot responses, and ultimately improving wider AI functionality.

The art of technology

Many, if not most, museum professionals encounter some kind of AI in their workplace. Computer vision can help with collections management, and robotic interpreters intrigue visitors.

And this study shows that AI has the potential to do much more to help manage a museums workload.

Fei-Fei Li and John Etchemendy, the Co-Directors of the Stanford Institute for Human-Centres AI stated that “If AI is to serve the collective needs of humanity, it must incorporate an understanding of what moves us — physically, intellectually and emotionally. It is critical that we design machine intelligence that can understand human language, feelings, intentions and behaviours, and interact with nuance and in multiple dimensions.” and ArtEmis offers an example of using responses to art to train models of emotional language.

In contemporary art, AI is being used by organisations like ART AI to create artworks that focus on evoking an emotional response. ‘As passionate believers in both art and technology,’ they say,  ‘we were greatly intrigued by the idea of developing an artificial intelligence that would be able to affect people emotionally, while inspiring creativity and innovative thinking.’

But these AI artists are trained through data sets of human made art and developed by human programmers. So should we consider the artwork to be made by the AI, or by the human originators of that AI? Or are they co-creators?

This train of thought applies equally to curation.

For the 2021 Liverpool Biennial’s publication Stages #9, Joasia Krysa and Leonardo Impett proposed an AI curated Biennial. ‘Our relationship to computers is rapidly changing and so are developments in automation (AI), and so is our understanding of creative practices, including curatorial practice.’ they wrote, ‘The overall project takes machine learning algorithms beyond the ‘search engine’ paradigm in which they have been mostly used to date, and instead considers them to be curatorial agents, working alongside human curators.’

And it’s not difficult to imagine a future where curation is assisted by AI.  A future where AIs can combine observations, quantitative, and emotional information to give museum staff powerful curatorial insights.

But how will this impact the visitor experience?

Who receives credit for AI-assisted work?

At best, AI offers a way to greatly enrich how we engage with our audiences, for example, through a highly trained chatbot. By engaging with culture we increase our sense of connection to others, and chatbots could synthesizing the input of a whole team of people.

At worst, robotics run a risk of becoming uncanny – creepy and a little bit too human – and turn our visitors off.

If AI can suggest an emotional response to art, it immediately seems more human. And, it’s tempting to use humanizing language about AI, which exaggerates its novelty and grabs our audience’s attention.

The work of constructing Dalí’s digital likeness offers a museum example of how human performance and AI-generated output combine in an interpretive character.

But, a study at MIT found that when humanizing language was used about AI artists, people were more likely to give them the credit for the artwork. Where they were not humanized, people instead gave credit to the team of people that use the AI for this task.

So, it seems that careful use of language around AI can help our visitors to embrace these experiences.  By reducing humanization of AI, we can reduce the risk of entering the uncanny valley (and make sure that the people behind the experience get the credit that they deserve).

AI in the museum toolkit

The ability of AI technology to learn about emotion offers greater possibilities for the interchange between art and AI.

For culture, AI challenges and enhances how we make, exhibit and understand art.

For AI, research can develop in a more meaningful and responsible way through learning from art.

And by acknowledging the importance of culture to describe human nature, AI research can become a powerful champion for the arts.