Abstract

Vienna’s Phonogrammarchiv is currently in the middle of the first migration of video data. The archive developed solutions for the challenging transfer procedures from mostly outdated but still operating equipment to modern hardware, and for linking legacy software for the playback process with today’s programs for recording and storage. Relying on rugged tape recorders backed by major manufacturers, and on a future-proof philosophy for the compatibility of playback devices over more than a decade turned out to be a good choice. Software solutions for parts of the migration process that replaced the originally expected hardware operations will be discussed, specifically the lossless transfer from a proprietary file format to more common or open standard formats.

The presentation will also point out workflow issues, recommending—at least for small archives—manual rather than automatic metadata processing for the first migration, done in tight cooperation of the archivist and the technician, in order to streamline the migrated data with respect to formats and metadata notations, and to correct a few errors in the archives database that inevitably occurred during the initial phase of the archiving process, when the workflow underwent occasional modifications. The primary goal for this admittedly time-consuming process is a mostly automatic second migration in the future.

Keywords

audiovisual archive, video department, technical issues, obsolete formats, analog sources, uncompressed file format, video data migration.

Introduction

Archiving of video footage at the Phonogrammarchiv in Vienna started in 2003 after an extensive investigation within the academic community (universities, museums, academy of sciences), covering holdings and properties—for example, formats—of video tapes ready for archiving.

The subsequent process of acquiring hard- and software for the archiving procedure followed three basic guidelines: quality, affordability and easy maintenance of the archiving system by an Austrian vendor. Therefore, the philosophy that has already been proven for audio archiving—analog sources being digitized in the best available process and stored in an uncompressed or lossless-compressed format, and digital sources nearly unchanged during processing and transfer to their final storage device—could also be applied, accordingly, to the video footage. Also, financial support was provided by the Academy of Sciences, and the local vendor guaranteed short response times in case of hard- or software problems with the archiving system.

The archive finally agreed to accept, temporarily, a proprietary, non-standard format for the archived video and audio files (DPS/DVA) because there was a safe loophole (SDI-connector) for the first migration to extract the standard format data sequence (ITU-601) included in the proprietary DPS/DVA container.

The workflow to be developed should also follow, as closely as possible, the task sequence already used for audio footage. However, modifications resulting from specific properties of the video stream, such as much higher data rates and larger files were inevitable.

According to recommendations, the start of the first migration was expected to be between six and ten years after the first archiving tasks, in order to switch from aging hardware to a new one, and to stay within the time frame of the final archiving media generations (LTO tapes) with backwards compatibility in reading the previous two generations.

The worldwide economic crisis at the end of the last decade affected Austria as well, and resulted in heavy cuts in personnel and budget, even threatening the existence of the archive for a while. So the start of the migration was delayed significantly.

Migration Workflow

Initial contemplations about the temporal structure of the migration offered many possible approaches ranging from a concentrated migration period of a few months, focusing almost all activities of the archive on the migration process, to a more or less continuous migration over the whole migration period of nearly a decade, and in parallel to the regular archiving tasks.

For future migrations, a fully automatic process certainly is a must. However, to ensure a smooth automatic data transfer in the future, it is advisable to correct, manually, all errors of the initial phase where workflow and technical issues had frequently been modified due to the lack of experience. Thus, the current procedures include a step-by-step transfer of the data, with both a technician and an archivist closely watching the process, correcting or complimenting data, and making relevant remarks for the archive’s internal database.

As an example, on some initial files, keyframes (as a support for a search in the archive’s database) were missing, so they have been added during the migration procedure. Also, some metadata describing scene changes were linked to displaced pointers at the timeline, because they were written down before the material was stripped from empty frames at the beginning of the file. These errors have been corrected, as well.

The technical issues were much more extensive. Back then, the nature of the archiving software suggested the time-saving storage of the original unaltered file together with an additional file that included a list of “virtual” files (representing the actual subfiles, had the original one been cut into “real” smaller parts). Time saving was of high priority due to the backlog of material ten years ago. Also, this constraint prevented us from creating integrity files. It simply took too long to calculate them. We accepted this status of limited data security—having only two copies of the archived files (two pieces of LTO-tape, of different manufacturers, stored in different places in Vienna).

Clearing backlogs are of less importance now. Consequently, the archive decided that, in parallel with the transfer from the proprietary DPS-container to a more standardized AVI-file, it was the right moment to create “real” files from the virtual ones. Furthermore, integrity checks will accompany the actual files from now on.

Hardware Issues

For the archiving process, our vendor supplied two good quality personal computers with extra dedicated hardware (to handle uncompressed files), and for the I/O-channels—for example, for the external striped hard disk arrays.

Naturally, after more than ten years of archiving there were signs of wear, so we replaced the heavily used hard disks with new ones. This was just in time, because now SCSI hard disks are more difficult to find on the market and more expensive. One SCSI controller failed during the initial migration phase and was replaced. The graphics card, unfortunately, belonged to a device family prone to failure after a couple of years because of an improper seal of the main chip, so it had to be replaced. The other components of the two computers still work without any problems.

Our final storage media for the first decade had been LTO tapes of generation 1 and 3. The three LTO-1 recorders had been heavily used, so when there was extra budget available we bought an additional LTO-2 that was available as a bargain when LTO-3 devices became the best-selling family of recorders, and another LTO-3 when LTO-4 and -5 were current products. Luckily, all of them operate flawlessly, with the LTO-3 recorders being slightly more sensitive to low humidity in the studio during the wintertime. The recorders did refuse to operate properly below 25% humidity, but stated the reason clearly on the error report. A small humidifier solved the problem.

As for the tapes themselves, among the few hundred tapes of LTO-1 and LTO-3, only three pieces were rejected initially by the recorder during the archiving process, and three others showed a reading error during migration, with two of them being capable of delivering the file by a second playback run. One single file could not be recovered, so we used the second copy for playback.

Software Issues

The original “archiving” software came together with the hardware ingest card of the PC. This was standard editing software, but was only used for virtual cutting of the original file and creating a description file with the transition points between the scenes. This piece of software had some minor issues that had never been corrected by updates, but we found workaround procedures to circumvent them and avoid the problems. One of these flaws was the susceptibility to rapid brightness changes in a scene, such as a flash, or on the three-second overlap period we initially provided to prove that no scenes or frames had been removed. We later omitted the overlap that stemmed from the analog times of archiving audio or video material.

For the first migration we decided to create real files, which takes some time and makes use of the copy command within the player window of the editing program. This feature is not typically used during the archiving process. Furthermore, we had to get used to another bug that prevented the system from creating files with more than about 44,000 frames. To this day, we do not know the reason for this behavior. It is definitively not a matter of internal storage space (RAM or hard disk), but there is a theory that it may be a driver problem with the outdated operating system, Windows 2000. Since most of the real files are smaller than that, it is not a big problem. We create larger files by calculating two smaller, slightly overlapping ones, and then glue them together with a frame-accurate open-source utility.

The original backup software that came with the operating system, Windows XP, was fairly stable during archiving, although it still contained a few harmless bugs. For example, compared to the first recorded copy, the recording time for the second copy of the LTO tape was significantly longer, making the procedure slower. However, rebooting before making the second copy solved the problem. Also, the software gave warnings when the tape was played back on another system, as opposed to the one it was actually recorded on. This was a particular issue when playing back multi-session recordings. This problem seemed to stem from operating systems that varied slightly because of minor update differences, but, fortunately, no files were lost.

Multi-session recordings had to be read back separately in order to ensure that the description was not overwritten on previously recorded files. This issue is preventable by paying attention during the migration process. Additionally, it was also a potential source of errors back then, because somebody could forget to store the correct description files. In that case, the original file was edited using the metadata from the archive’s database which accurately listed the timestamps of the scenes. This process, of course, is extra time-consuming work.

After the rendering of “real” files, pairs of DPS and DVA Files, we used a commercially available format converter that combined the contents of the DPS and DVA container to one single AVI-file. Also, we made use of a filter provided by the DPS community, downloading it from the webpage of the company that later on acquired the DPS/DVA vendor. This process is now a fairly quick procedure. In the case of problems with the software conversion, we still could make use of the ITU-601-output on the SDI-terminal of the DPS-System as a base for further processing.

There are occasional mistakes to be corrected, such as denomination, typing errors in the database or on the LTO-cassette leaflet, missing keyframes, or incorrectly rendered “browsing” files with wrong field order. Therefore, the transfer to the new storage media, LTO-5 tape, includes a careful check of these issues against the archive’s database.

The original backup software is no longer supported on systems using Windows 7 and upwards, so we switched to the new linear tape file system, known as LTFS-format, and used command line software and drivers provided by the tape recorder manufacturers. This process is a relatively easy job. In contrast to the archiving process from years ago, these migration recordings, as well as all future recordings during the archiving process, using LTFS will be done by the system operators in order to correctly mount the recorder and provide an adequately fast data stream for the recorder from the hard disk array. The huge amount of storage space of LTO-5 (1.5 TB) requires an intelligent preparation of the sum of projects to be recorded on tape, preferably in one session, and directly from an internal hard disk over a SAN-connection (LTO-1 tapes could be properly fed with data from an external hard disk using the popular but slower USB2-connection).

Recommendations

For the first migration, we recommend an admittedly time-consuming, but in the long run, rewarding manual migration in tight cooperation between an archivist and a technician. This process applies, particularly, to material created during the initial phase of archiving, where workflow, formats, and equipment may have been modified frequently, and when a more uniform structure of the stored material is highly desirable.

Since migration usually has to include original hard- and software, it pays to keep the systems in good condition with occasional checks. Two complete sets of archiving computers, plus backup tape recorders, or other storage devices are recommended. It is also advisable to get the latest possible updates and drivers for backup- and editing software, and to buy some spare parts as long as they are available (e.g., SCSI-controller).

If LTO or similar systems are used, it is also a good idea to acquire additional recorders that offer backward-reading compatibility.

In conclusion, video data migration should, in our opinion, be seen as a continuous process in parallel to the regular archiving tasks. It should be repeated within a period of six to eight years, thus having the fully functional hard- and software technologies that enable a smooth transition from one storage media generation to the next one.

FRANZ PAVUZA has been with the Phonogrammarchiv of the Austrian Academy of Sciences since 2002 and is responsible for the installation and operation of the videographic section (technical issues). He has an master’s degree in electronics from the Vienna University of Technology and until 2002 worked at the university as an R & D assistant for projects in cooperation with industry partners and government-supported funds in audio, video (basic research, applications in test & measurement) and industrial electronics (development of prototypes). He is a member of the AMIA (Association of Moving Image Archivists) and the German FKTG (Fernseh- und Kinotechnische Gesellschaft).

His job at the Phonogrammarchiv includes presentations for training sessions and workshops (ERPANET, National Library of Austria) and conferences (AMIA, IASA, FIAT, DELOS).

He currently supervises the first major video data migration at the Phonogrammarchiv.