How Museums Are Blending Digital and Physical to Engage Visitors Like Never Before

Imagine walking into a museum where art literally grows before your eyes or where your actions can direct virtual water flowing through a digital landscape.

Four projects discussed at the Museums, Games & Play Summit 2024 show different ways to connect tactile controls with digital responses: Art Forest, Muck Hunt!, Grabbit and Paint ’n’ Play. Their practical lessons concern scale, durability, prototyping and shared use.

Combining physical controls and digital responses

Physical-digital games combine a screen or digital system with something visitors can handle, move or control. Arcades provide familiar examples, from steering wheels to buttons. In museums, the useful design question is what a physical interaction enables visitors to do with the content.

The speakers questioned how much visitors wanted another phone-based or touchscreen experience and explored alternatives using tactile controls. These observations informed their projects; they do not establish a preference shared by every visitor or age group.

Art Forest: experimenting at a manageable scale

One interesting example is “Art Forest,” a project by Jeff Steward, Director of Digital Infrastructure and Emerging Technology at Harvard Art Museums. “Art Forest” began as a purely digital game presented at a student event in 2019, where participants interacted via a screen and mouse. However, as the project evolved, it morphed into a more dynamic phygital experience, incorporating physical controllers made from cardboard boxes filled with buttons, knobs, switches, and sensors.

This shift not only enhanced the interaction but also introduced an element of unpredictability, making each session unique. In “Art Forest,” data from art objects is transformed into digital trees that grow within a shared virtual forest, creating a metaphorical ecosystem that reflects the museum’s collections. The physical controllers give visitors the power to influence the environment, such as adding wind to make the trees sway, bridging the gap between the physical and digital worlds.

Steward describes Art Forest as an experiment suited to a teaching museum and relatively small groups. It invites slow observation and exploration, with a design that can change between installations. The cardboard controllers are part of that experimental scale, rather than a tested answer for a high-volume gallery.

Muck Hunt!: choosing controls that survive real conditions

Chris Evans, founder of Drumminhands Design, developed “Muck Hunt!”—an interactive game designed for the Minnesota State Fair. The game had to withstand heavy use at the fair while delivering its message about cleaning invasive species from outdoor equipment. Initially, “Muck Hunt!” used infrared light guns to detect player actions, but sunlight interference led to a pivot towards using Phidgets, and later, trackballs for player input.

Although the shift to trackballs might seem like a step back from the original vision, Evans notes that this simple solution proved to be more durable and reliable in the high-traffic environment. “Muck Hunt!” teaches players the correct tools and methods to clean off invasive species, combining learning with hands-on fun. Evans reported conversations about invasive species alongside high use of the game; the account does not quantify learning gains.

Evans’s approach emphasizes the importance of stepping away from screens and creating tactile, hands-on interactions. “People are surrounded by screens. They carry screens in their pockets all the time. Give them something different, something that’s really tactile. That’s gonna get people’s attention,” he explains, highlighting how physical-digital experiences can break through the noise and create meaningful engagement.

Grabbit: building a playable prototype in 30 days

Kellian Adams Pletcher, Director of G.L.A.M. Innovation at FableVision Studios, brings a fresh perspective on rapid prototyping. Over her 15-year career, Pletcher has developed numerous interactive experiences for museums, demonstrating that phygital projects do not always require extensive resources or timelines. By setting clear parameters—goals, resources, and restrictions—Pletcher’s team has delivered fully functional prototypes in as little as 30 days.

One example is “Grabbit,” developed for the Rietberg Museum in Switzerland, which uses RFID tags to create a physical-digital quiz game for children. The game underwent rapid prototyping stages, initially using basic materials like cardboard and foam, and was being developed into a full interactive exhibit at the time of the presentation. Pletcher’s approach shows that even with limited budgets, museums can create engaging phygital experiences by embracing a prototype mindset.

Paint ‘n’ Play: designing for children together

“Brushter” was a physical-digital art tool for children developed by the National Gallery of Art. While popular, it lacked robust data tracking. In response, the team led by Robyn Bortz conducted a comprehensive usability study in 2022, leading to the game evolving into “Paint ‘n’ Play,” a multi-touch, interactive art experience designed to foster social interaction and creativity among children and their caregivers.

At the National Gallery of Art, the useful question was not simply whether children liked Brushter. Observation showed that its single-user design got in the way of the shared experience children and caregivers wanted. Paint ‘n’ Play therefore supported several touch points and made clearer connections to works in the galleries. That is a practical brief for an interactive: identify the moment that frustrates visitors, change the design and test whether the new version lets them do what they came to do.

Testing the interaction before scaling it

These examples suggest a sequence for testing an interactive: identify the visitor action, build a manageable prototype, observe shared use and check the controls under real operating conditions. As Pletcher puts it, “Set yourself a time, set yourself a date. Try it out. See what you can build within your own resources, goals, and restrictions.”

The right scale will differ between a small-group experiment and a busy public installation. Plan maintenance and testing around the setting in which the game will actually be used.

For a related perspective, explore mapping a museum game’s whole visitor journey.

For a related perspective, explore testing a game inside the Rijksmuseum.

This article is based on the presentation “Building Games with Digital-Physical Interfaces” delivered by Kellian Adams Pletcher (FableVision Studios’), Robyn Bortz (National Gallery of Art), Chris Evans (Drumminhands Design) and Jeff Steward (Harvard Art Museums) at the Museums, Games & Play Summit 2024.