Abstract

In the PREFORMA project, collection holders pool their resources to work with experts at small- and medium-sized companies. This collaboration is made possible through a “pre-commercial procurement” approach, funded by the European Commission and memory institutions across the continent. After a selection and early design process, the project awarded MediaArea, who previously built the widely used MediaInfo application, the contract to: (1) do standardization work on an open-source format for video, and (2) develop a toolset that performs policy checks and corrects remaining errors in the build-up of incoming or existing files. This paper gives an overview of the format selection process, the development work that has gone into building the MediaConch toolset, and the standardization activity of the selected preservation format for audiovisual assets.

Keywords

digital preservation strategy, OAIS model, conformance checking, policy checking, normalization, file format selection, open file formats, audiovisual archives, open-source software, standardization process.

Introduction

In the PREFORMA project, collection holders pool their resources to work with experts at small and medium-sized companies (or SMEs). This collaboration is made possible through a “pre-commercial procurement” approach, cofunded by the European Commission and memory institutions across the continent. The project seeks to develop solutions for three file types: documents, still images, and audiovisual materials. After a selection and early design process, it awarded MediaArea, who previously built the widely used MediaInfo application,¹ the contract to: (1) do standardization work on an open-source format for video, and (2) develop a toolset that performs policy checks and corrects remaining errors in the build-up of incoming or existing files. This paper gives an overview of the format selection process, the development work that has gone into the building of the MediaConch toolset, and the standardization activity of the selected audiovisual preservation format.

The PREFORMA Challenge

A file format is a standardized way of encoding information for storage. Pragmatically, different software applications and versions can vary in the exact ways they implement or apply a certain standard. Regarding long-term preservation efforts, memory institutions should have full control of the process for testing the conformity of files to these standards before they are ingested into their archives.

The conformance checker software developed in the PREFORMA project is intended for use within the Open Archival Information System (OAIS) Reference Framework. It focuses in particular on the OAIS ingest function, which represents the first step in the digital curation and preservation workflow. PREFORMA aims to work on one specific step in the OAIS process, that is, file validation. The tools it is tendered for are developed in an iterative process, with multiple releases, and with a number of experiments with ‘real’ data sets (files) from memory institutions.

One of the most important technical challenges institutions and organizations that focus on long-term storage face, is the tactic of normalizing the files within their collections. When an organization decides to limit the number of file types it takes responsibility for, it makes managing these files a less haphazard task. Homogeneous collections allow for bulk monitoring and bulk solutions. A conformance checker can help by narrowing down the variety of files for ingest. For the PREFORMA project, a conformance checker should:

1. verify whether a file has been produced according to the specifications of a standard file format;

2. verify whether a file matches the acceptance criteria for long-term preservation by the memory institution;

3. report, in human and machine readable format, which properties deviate from the standard specification and acceptance criteria; and

4. perform automated fixes for simple deviations in the metadata of the preservation file.

A memory institution needs to know the files they create or receive are healthy. Validation tools help control whether composition is correct. If a file is malformed upon arrival, this must be known upon ingest. Whether the institution’s policies say that the error should be corrected (and thus change the original), or note the flaw in the metadata, a memory institution needs tools to aid this purpose. The main objective of the PREFORMA project is to give memory institutions these types of tools—by developing a toolset that enables the testing process to happen within an iteration fully under the memory institution’s control.

PREFORMA’s research and development activities aim to empower memory institutions to gain full control over the technical properties of preservation files. This is achieved through the development of an open-source conformance checker and the establishment of a healthy ecosystem around an open-source ‘reference’ implementation of three specific file formats.

Selecting an Audiovisual File Format

The audiovisual files that memory institutions receive need to be highly adapted to the (post)production and distribution workflow of the domain in which they are produced. Files from broadcasters are made for streaming and efficient editing. Files created for film productions have extensive functionality for encrypted access to the content. These properties often complicate and weaken the long-term preservation of the content. Workflows in audiovisual collections mostly rely on professional broadcast and film production software for efficient and reliable processing of audiovisual files. Consequently, collections prefer using the corresponding proprietary formats for managing these files.

For PREFORMA, fitness for long-term preservation depended on the following four criteria:

1. A format that is lossless. The format should be able to capture content in an uncompressed or mathematically lossless encoding, and retain as many original properties as possible (i.e., bit-depth, resolution, etc.). Not losing information in compression is a requirement for implementing an effective preservation strategy. This condition meant ruling out well-adopted but lossy codecs.

Based on experience from digitization projects, PREFORMA considered the advantages of lossless compression for storage cost and self-description. While storage cost is likely a short-term concern, in the project’s start-up phase there was still a clear financial concern on the part of decision-makers. Uncompressed would complicate adoption for a lot of collections where storage costs often represent a big argument in favor of choosing a lossy format. Beyond storage, lossless compression gives advantages for self-description and fixity (frame checksums, error concealment, etc.). FFV1’s version 3 has, amongst others features, integrity validation via CRC checksums, which is highly beneficial for long-term preservation.

2. A format that is Free/Libre. This requirement was a key issue for the PREFORMA project. Software developed within the scope of the project must be licensed under a specific copyleft license (i.e., “GPLv3 or later, and MPLv2 or later” [Lemmens, 2014]) to ensure that the tools for checking archive files will be available over very long periods. This requirement ruled out the MXF wrapper since the current licensing of the required MXF specifications do not allow for implementing a validator under the given license. Moreover, the specifications for MXF are behind a paywall and thus limit their collaborative use. The cost for acquiring necessary specifications and access to the standardization process of MXF was too high for an open-source project, in particular in comparison with MKV. Furthermore, PREFORMA could not tender for MXF because of Swedish Law that obliges public services to tender for open formats as defined in the European Interoperability Framework (European Communities, 2004).

3. A format that is well documented. There needed to be a specification available, which was additionally accessible and well maintained. Without it, writing an implementation or creating test files becomes a challenge. PREFORMA considered that working on the documentation and standardization during the course of the project would be more feasible with the MKV community than with SMPTE for MXF. Given that there was already a validator available for MKV, the project expected that the investment in this container would have a higher payoff.

4. A format that has been adopted. Some formats have very promising specifications, but the project needed a degree of adoption that proved that the file format is viable and usable in real life situations.

In the end, the project opted for standards where it could imagine that the existing shortcomings could be solved within the scope of the project. It evaluated the tenders in two stages and chose the project that offered the best strategy for solving the existing shortcomings. The result of this exercise for the audiovisual format ended up in a set of two containers (MKV and Ogg) and three video/image codecs (JPEG2000, FFV1, and Dirac). Tenderers were invited to make an R&D project proposal for a combination of a container and codec from this list. For long-term preservation of digital video, choosing both a sustainable container format as well as audio/video codecs are necessary. There is no consensus to date among the archival community as to which file format or codecs should be used for preservation purposes for digital video. The previously proclaimed encodings were JPEG 2000 (lossless) and uncompressed video. FFV1 turned out to be a viable addition to that choice and was, therefore, recently added as a suitable option for preservation encoding.

Professional archives are already using FFV1, with compression ratios comparable to JPEG 2000 lossless and its lower computing requirements, as their long-term storage codec.² Especially, archives whose collections do not feature extensive broadcast materials, but rather oral history- or ethnographic-recordings and the likes, “favored the lossless FFV1 encoding” in communications with the “Federal Agencies Digitization Guidelines Initiative” (FADGI) team (Fleischhauer, 2014). FFVI is also listed as a format option for long-term preservation of moving images on sites of the US Library of Congress, State Records NSW, and others. The Society of American Archivists published a paper in August 2014 suggesting only FFV1 as preservation codec for video. The Digital Preservation project at the US Library of Congress identified AVI and Matroska as common container formats for FFV1.³

The MediaConch Application

The first design phase lasted four months. Six suppliers were selected (two for each media type), and the suppliers of the three best designs were requested to proceed to the current prototyping phase. The MediaArea team has worked on several software tools used in preservation data conformance, including MediaInfo, a tool to extract metadata from audio and video files, and Bay Area Video Coalition’s QCTools project, a tool that performs quality control checks and analysis on digitized analog video.

MediaArea is one of three teams that have moved forward from Phase 1 (design) into Phase 2 (software prototyping). This second phase will last twenty-two months. In Phase 1, MediaArea was tasked with researching community standards and developing file conformance checks for FFV1, Matroska, and LPCM to support efforts of long-term preservation in memory institutions. In Phase 2, MediaArea has been developing MediaConch, an extensible, open-source software project consisting of an implementation checker, policy checker, reporter, and fixer that target preservation-level audiovisual files. It provides detailed and batch-level file conformance checking via an adaptable and flexible application program interface that is accessible by the command line, a graphical user interface, or a web-based shell. MediaConch, furthermore, makes use of a new reporting feature for documenting the structure and contents of digital files with a particular concentration on audiovisual data: MediaTrace. It is a complementary tool to MediaInfo that summarizes a file’s significant characteristics and offers comprehensive documentation of file information in a powerful XML format for precision identification and conformance checking.

The CELLAR Working Group

The goal of the PREFORMA project is to address the challenge of implementing good quality standardized file formats for preserving data content in the long term. Its objective in regards to standardization FFV1 and Matroska is to improve the sustainability factors for both formats—including disclosure, credibility, and transparency—while also creating a mechanism to authoritatively clarify the intent of the specifications. The MediaArea team is dedicated to the further development of the standardization of the Matroska and FFV1 formats to ensure their longevity as a recommended digital preservation file format.

After presenting on FFV1 and Matroska at IETF 94 in 2015, they have incorporated feedback and input from IETF members to draft a new charter for a working group that includes FFV1, Matroska, and FLAC under the name CELLAR: Codec Encoding for LossLess Archiving and Real-Time Transmission. Like all the Task Force’s working groups, it works in an open sense—by means of mailing lists and open gatherings.

FFV1 is a lossless video codec, and Matroska is an extensible media container based on EBML (Extensible Binary Meta Language), a binary XML format. There are open-source implementations of both formats, and an increasing interest in and support for the use of FFV1 and Matroska. However, there are concerns about the sustainability and credibility of existing specifications for the long-term use of these formats. These existing specifications require a broader review and formalization in order to encourage widespread adoption.

There is also a need for a lossless audio format to complement the lossless video codec and container format. FLAC is a lossless audio codec that has seen widespread adoption in a number of different applications, including archival applications. While there are open-source implementations of the codec, no formal standards for either the codec itself or its use in container formats currently exist. Review and formalization of the FLAC codec standard and its use in Matroska container formats are needed for wider adoption.

Using existing work done by the development communities of Matroska, FFV1, and FLAC, the Working Group will formalize specifications for these open and lossless formats. In order to provide authoritative, standardized specifications for users and developers, the Working Group will seek consensus throughout the process of refining and formalizing these standards. The Working Group seeks consensus and refinements for specifications for both FFV1 and Matroska in order to provide authoritative, standardized specifications for users and developers.

ERWIN VERBRUGGEN is project lead at the Research & Development department of the Netherlands Institute for Sound and Vision where he works on research projects related to preservation and access in audiovisual archives.

DAVE RICE is an audiovisual archivist and technologist, whose work focuses on independent media, open-source technology in preservation applications, and quality control analytics. He is the project lead for the MediaConch team.

BERT LEMMENS works at the Belgian Centre of Expertise PACKED, where he is a specialist in preservation tools and semantic applications for cultural heritage. He led the task of gathering the project requirements for the PREFORMA tender specification.

JéRôME MARTINEZ is the Founder and President of MediaArea.net, the lead developer of MediaInfo, and a technical consultant to a variety of projects within the fields of broadcast video, audiovisual archiving, and web video.

ASHLEY BLEWER is a developer, archivist, moving image specialist, applications developer at the New York Public Library, and open-source enthusiast, and has worked on BAVC’s QC Tools and the MediaConch Project.

EMANUEL LORRAIN works for PACKED, a Centre of Expertise in Digital Heritage created in 2005, based in Brussels. He works as a researcher developing and disseminating knowledge about the preservation of audiovisual heritage and media artworks.

ANTONELLA FRESA is an ICT expert, and Director and General Manager at Promoter SRL. She is Technical Coordinator of PREFORMA and Communication Manager of national and European projects in the domains of digital cultural heritage, creativity, and cocreation, citizen science, smart cities, digital preservation and e-Infrastructures.

CLAUDIO PRANDONI works at Promoter SRL, where he is involved in several projects supported by the European Commission, especially targeted at the tourism and cultural heritage sectors. He is the assistant to the Communication Manager of PREFORMA.

Notes

1.  MediaInfo. Retrieved July 4, 2016, from https://mediaarea.net/en/MediaInfo.

2.  An up-to-date list of institutions can be found on the FFV1 Wikipedia page: https://en.wikipedia.org/wiki/FFV1#List_of_institutions_known_to_use_FFV1 (accessed 2016/10/07).

3.  http://www.digitalpreservation.gov/formats/fdd/fdd000349.shtml; http://www.digitalpreservation.gov/formats/fdd/fdd000343.shtml.

Bibliography

European Commission. (2013, June 25). Communication from the commission to the European parliament, the council, the European economic and social committee and the committee of the regions. Against lock-in: building open ICT systems by making better use of standards in public procurement. European Commission. Brussels, BE. Retrieved from https://ec.europa.eu/digital-single-market/en/news/against-lock-building-open-ict-systems-making-better-use-standards-public.

European Communities. (2004). European interoperability framework for pan-European eGovernment services [internet]. European Commission. Luxembourg: Office for Official Publications of the European Communities. Retrieved from http://ec.europa.eu/idabc/servlets/Docd552.pdf?id=19529.

Fleischhauer, C. (2014, December 3). Comparing formats for video digitization [webpage]. The Signal Digital Preservation. Retrieved from https://blogs.loc.gov/digitalpreservation/2014/12/comparing-formats-for-video-digitization/.

Fresa, A., Justrell, B., & Prandoni, C. (2015). Digital curation and quality standards for memory institutions: PREFORMA research project. Archival Science, 15(2), 191–216. Retrieved from http://doi.org/10.1007/s10502-015-9242-8.

Lemmens, B. (2014). Challenge brief (artefact document) (p. 15). Retrieved from http://www.digitalmeetsculture.net/wp-content/uploads/2014/06/PREFORMA_Challenge-Brief_v1.0.pdf.

Murray, K. (2015, September 23). Improving technical options for audiovisual collections through the PREFORMA project [web page]. The Signal Digital Preservation. Retrieved from https://blogs.loc.gov/digitalpreservation/2015/09/improving-technical-options-for-audiovisual-collections-through-the-preforma-project/.

Society of American Archivists. (2014). Museum archives section standards and best practices working group electronic records project. Retrieved from http://www2.archivists.org/sites/all/files/MAS%20E-rec%20project.pdf.