When optical discs exceed 8 TB

What if optical discs exceed 8 TB?

 

When optical discs exceed 8 TB in future, we might have a problem due to the limitations of the UDF file system.

 

## The birth defect of UDF

 

While UDF uses 64-bit file sizes, it uses 32-bit cluster numbers. This might seem like a poor choice today, but remember, UDF was created in the 1990s for 4.7 GB DVDs.

 

Because the sector size on optical discs is 2048 bytes, UDF uses it as its cluster size too, and limits the volume size to 2048×2^32 bytes, hence 8 TB. Without this limitation, UDF could store files in the exabyte range. That's more than the size of the Internet Archive (around 0.2 exabytes).

 

## The 32-bit handicap in other places

 

32 bits cause 4 GB limits in various places. 32 bits are also the root of the year 2038 bug

 

32 bits are the thing that limits FAT32 file sizes to 4 GB. Granted, FAT32 was made in the 1980s, so 4 GB were seen as astronomical and out of reach. Now it seems tiny, though most files people store are still under 4 GB (pictures, music, videos that are not too long).

 

Depending on bitrate, 30fps 4K video reaches 4 GB after 10 to 15 minutes, which is still quite a lot. But FAT32 is obviously useless for storing stuff like disk images, except if you are happy to fumble around with split files.

 

At least, FAT32 usage will outlive ext2 and ext3 because FAT32 is the common denominator across operating systems and multimedia appliances with USB ports (DVD players, car audio systems, etc.), and most especially because it is unaffected by the year 2038 bug. And that bug is also 32-bit-related because it is caused by a 32-bit Unix epoch second counter!

 

32-bit systems are also limited to 4 GB of total memory, meaning RAM + VRAM. 4 GB are barely useable nowadays, given how bloated websites have become. One YouTube tab might set you back by 0.6 GB of RAM rather than a few dozen megabytes.

 

In today's computing world, four billion anything are simply too few.

 

Therefore, remember, **nothing with 32 bits is future-proof.**

 

# Optical discs can contain exFAT

 

Any file system can be written to optical media, including exFAT, NTFS, ext4, ZFS. Nothing technically prevents those file systems from being written to write-once optical discs, however, they can not be modified once written.

 

Some non-Linux systems might not look for non-optical file systems on an optical disc and not recognize the disc. The entire image also has to be written sequentially in one go, so you can not add some files today and some new files tomorrow.

 

File systems created for random write data storage (USB sticks, HDD) expect to be able to modify the table of content (which contains all file and folder names) at the beginning when adding data. Only UDF and ISO9660 support rewriting the table of content at a new location through multisessioning, and only UDF can do packet writing, which allows the modification of files in real-time like on a flash drive by simulating a writable volume.

 

This means an exFAT image has to be created on an external hard drive that matches the size, then filled up completely (any unused space is wasted), then written to the optical disc. Or perhaps the hard drive needs to be formatted in exFAT and then filled up directly. Since Unix uses the "everything is a file" philosophy, the device file of the external hard drive (stored inside /dev/ ) can be treated as an image file.

 

For over 8 TB optical discs in future, I would recommend exFAT because it is patent-free and non-proprietary since 2019, and widely supported across platforms. Microsoft graciously open-sourced exFAT in 2019, though it was already reverse-engineered as early as 2010. exFAT doesn't have journaling, but that's not needed for an image written once and then never modified again.