How AI has learned human emotions from art

Researchers at Stanford University have programmed their AI algorithm to look at a work of art, and then, astonishingly, to form an emotional response to it.

More than this, the AI can form multiple responses.  for different parts of the picture. This means that is can read different parts of the picture, understand emotional responses from the viewpoints of different people, and in different contexts.

So, has art taught an AI algorithm empathy?

And what might this mean for museums in the future?

Meet ArtEmis

The study conducted by Stanford University’s Institute for Human-Centered AI trained their artificial intelligence, called ArtEmis, to read emotional responses from great artworks.

By drawing from a dataset of over 81,000 paintings and 440,000 human responses, the AI algorithm learned to predict how a human might respond to the artwork – and then to justify this prediction in words. Empathy is built in.

For Van Gogh’s Starry Night, the AI responded with ‘Awe. The blue and white colours of this painting make me feel like I am looking at a dream’.

When shown Dali’s The Persistence of Memory, ArtEmis replied with ‘Fear. It looks like an animal is laying dead on the ground’.

These responses go further to understand images than AI has gone before. And it’s truly extraordinary that the AI has the ability to respond differently to different parts of a work, and recognize that human responses may differ, too.

AI does not have a great reputation for its emotional intelligence. Emotion recognition technology is on the rise and has masses of potential, from marketeers tracking facial responses to law courts using AI to decipher the expressions of those standing trial. However, it’s also fraught with issues such as extremely troubling racial bias, simplification of psychological states, and dehumanization. This last has been termed ‘emojification’ by Dove Tail Labs in their Emotion Recognition System, which seeks to open up discourse around this technology (and the website has a brilliant – and worrying – AR camera to try yourself).

So, the complex understanding that ArtEmis offers is a leap forward in bestowing emotional intelligence on our AI.

And the creators can see a myriad of uses for it, from helping designers to test the effectiveness of their work, to enriching chatbot responses, and ultimately improving wider AI functionality.

The art of technology

Many, if not most, museum professionals encounter some kind of AI in their workplace. Computer vision can help with collections management, and robotic interpreters intrigue visitors.

And this study shows that AI has the potential to do much more to help manage a museums workload.

Fei-Fei Li and John Etchemendy, the Co-Directors of the Stanford Institute for Human-Centres AI stated that “If AI is to serve the collective needs of humanity, it must incorporate an understanding of what moves us — physically, intellectually and emotionally. It is critical that we design machine intelligence that can understand human language, feelings, intentions and behaviours, and interact with nuance and in multiple dimensions.” and ArtEmis shows that art is the ideal vehicle for training AI in human feelings.

In contemporary art, AI is being used by organisations like ART AI to create artworks that focus on evoking an emotional response. ‘As passionate believers in both art and technology,’ they say,  ‘we were greatly intrigued by the idea of developing an artificial intelligence that would be able to affect people emotionally, while inspiring creativity and innovative thinking.’

But these AI artists are trained through data sets of human made art and developed by human programmers. So should we consider the artwork to be made by the AI, or by the human originators of that AI? Or are they co-creators?

This train of thought applies equally to curation.

For the 2021 Liverpool Biennial’s publication Stages #9, Joasia Krysa and Leonardo Impett proposed an AI curated Biennial. ‘Our relationship to computers is rapidly changing and so are developments in automation (AI), and so is our understanding of creative practices, including curatorial practice.’ they wrote, ‘The overall project takes machine learning algorithms beyond the ‘search engine’ paradigm in which they have been mostly used to date, and instead considers them to be curatorial agents, working alongside human curators.’

And it’s not difficult to imagine a future where curation is assisted by AI.  A future where AIs can combine observations, quantitative, and emotional information to give museum staff powerful curatorial insights.

But how will this impact the visitor experience?

Avoiding the uncanny

At best, AI offers a way to greatly enrich how we engage with our audiences, for example, through a highly trained chatbot. By engaging with culture we increase our sense of connection to others, and chatbots could synthesizing the input of a whole team of people.

At worst, robotics run a risk of becoming uncanny – creepy and a little bit too human – and turn our visitors off.

If AI can suggest an emotional response to art, it immediately seems more human. And, it’s tempting to use humanizing language about AI, which exaggerates its novelty and grabs our audience’s attention.

But, a study at MIT found that when humanizing language was used about AI artists, people were more likely to give them the credit for the artwork. Where they were not humanized, people instead gave credit to the team of people that use the AI for this task.

So, it seems that careful use of language around AI can help our visitors to embrace these experiences.  By reducing humanization of AI, we can reduce the risk of entering the uncanny valley (and make sure that the people behind the experience get the credit that they deserve).

AI in the museum toolkit

The ability of AI technology to learn about emotion offers greater possibilities for the interchange between art and AI.

For culture, AI challenges and enhances how we make, exhibit and understand art.

For AI, research can develop in a more meaningful and responsible way through learning from art.

And by acknowledging the importance of culture to describe human nature, AI research can become a powerful champion for the arts.