How NHM Built Storage Infrastructure for Millions of Digital Records

In January 2022, the Natural History Museum reached five million digital records available through its data portal. This 2022 conference account also reported 32 billion record downloads in over half a million download events since 2015. Published records, record downloads and download events are different measures: records can be downloaded repeatedly, and one event can contain many records.

At the MuseumNext Digital Summit 2022 both programme manager of RECODE, Steen Dupont and London Technical Solutions Architect, Paul Ward from the Natural History Museum shared their journey to reaching 5 million digital records and the challenges it presented to the wider museum IT infrastructure.

The storage problem behind a public data portal

Infrastructure is hugely important when considering the digitisation of collections. In this case, it refers to networks, servers, storage, backup, disaster recovery, and the collections management system itself. Without infrastructure, collections are likely to be located on media files on local hard drives and data in individual spreadsheets and databases.

The speakers described a specimen file being copied up to nine times across the museum’s IT systems and backup processes, including tape. That copy count should be distinguished from storage tiering, which concerns how data is placed across storage tiers. The items sent to tape are essentially being sent to storage due to inactivity and it can take up to 90 seconds to retrieve any items sent to tape.  Fine for individual requests, but not ideal for large-scale data downloads.

The speakers reported an estimate of up to six months for recovery after a malicious attack or disaster in the storage context they described. It was in this framework that a new approach to the museum infrastructure was sought with new Infrastructure initiatives moved forward.
Serving different audiences from a shared asset library adds publication and access decisions to the storage plan.

Managing files from ingest to access

FLOWS

FLOWS is the first of the infrastructure initiatives created by the museum in response to the perceived challenges it faced. FLOWS tracks files, workstation, has server processes, an automatic copy clear-down and issues reporting. Allowing the team to automatically remove ingest artefacts at the beginning of the copying process.

Planning retrieval and recovery

Valentine

The speakers described Valentine as a second infrastructure initiative intended to change archival storage and improve retrieval. Changing storage media does not by itself guarantee that data will remain intact. As described in 2022, it stores data in a cloud object storage system instead of tape. When a file is saved to Valentine, it is saved to identical storage systems in two different data centres. Within 15 minutes, the file is erasure-coded with redundant data and split into three parts. Each goes to a separate data centre.

Again, if a file is not accessed for a period of time, it is removed from the initial tier, leaving stub files. The speakers reported access to a migrated file in under two seconds.

RECODE

RECODE refers to a programme for rethinking collections data and the procedures around it. NHM’s stakeholder-led collections data work addresses the related question of how systems should fit curatorial and research workflows.

RECODE is about finding a new collections management system that will bring the museum into the future using a stakeholder-first approach.  After all, it is stakeholders who know what the system needs to do and it is up to the programme team to have those long and in-depth conversations to understand that.  But it’s also about how the museum processes a specimen and how to go from acquisition, or acquiring a collection, to actually loaning a part of that out.

RECODE is hugely important as it is creating a system that is flexible, configurable and manageable by the institution. The institution has the knowledge about the sector, and the programme team want to drive whatever the technology is to fit to that.

The number of published records is only one part of the workload. A digitisation service also needs to move files, verify that copies remain intact, retrieve older material and recover after a failure. Those are separate tasks with separate performance targets; faster access to one archived file does not establish how quickly the whole service can be restored.

Steen Dupont and Paul Ward from the Natural History Museum spoke at the MuseumNext Digital Summit in June 2022.

–

MuseumNext offer online learning for museum professionals striving for engaging, relevant and flexible professional growth content. Learn more about our virtual museum conferences here.