The massive problem of data lock-in

Hendrix7 · Mar 1, 2024

Imagine that the camera software on your smartphone did not store your pictures and video recordings in DCIM/Camera but instead into a private, locked-in directory such as /data/… , which normally is inaccessible from the outside.

This means that no other application can view the photos and videos you have taken, and you can not move them to a computer or USB stick or hard drive or any other external device.

Imagine you could only view your photographs and play back your videos through the stock gallery app that came with the smartphone. If you ran out of space, you could not move your pictures and videos out. Once you fill up the 128 GB (or however much) of internal storage that your phone comes with, that's it. You would have to delete older photos to make space.

Imagine you could not back up your data anywhere else. When your smartphone breaks or stops working for whichever reason, there go your memories.

Does this nightmare sound familiar?

Let me introduce you to the data lock-in problem.

We have accepted data lock-in on mobile web browsers and messengers.

Have you noticed that none of the major mobile web browsers offer a built-in way to export your bookmarks and history and session (list of open tabs) to a file?

The leading mobile web browser, Chrome, only stores three months of browsing history.

Why would you want more than that? If you remember a page you have visited long ago, it is much easier to find it through browsing history than through web search, since the Internet is flooded with an avalanche of new information, and older posts and videos sink deep in the search results, or might have been removed. If a video has been taken down, you do not want to be searching for something that does not exist anymore through the web search. It is better to see immediately it was removed than "searching in the dark" for something that can not be found.

Also, you might want to export your current session to a file so you can close all tabs and start afresh without losing your current session. The ability to export tabs prevents "tab hoarding", since you know you can get the previous session from the exported file anytime. The current mobile browsers do not have this simple feature.

Currently, the only way to export tabs on a mobile web browser is to tediously manually copy the URL of each opened tab into a text file. For several hundreds of tabs, this takes hours to finish. A simple tab export feature would do the same work within seconds.

On desktop web browsers, exporting user data can be easily achieved through using extensions, but mobile web browsers largely do not support extensions. The one that does support extensions, Kiwi Browser, depends on the Google Chrome Extension store, which has added a login requirement in 2023. Extensions "sideloaded" through a CRX file can not export user data "for security reasons™ ".

The same applies to messaging apps. Do the pre-installed messaging apps of Google and Samsung let you save your messages into a text file? No. The only way to get your messages out is to copy them individually. If you have over 9000 messages, good luck with that.

I, for one, would love to see a law requiring web browsers and messaging apps to have export functionality. At least Microsoft set an example by recently adding history exporting to their Edge desktop browser, but since the browser profile folder is accessible on desktop anyway, and extensions for exporting history existed, it made little difference. It would have made a significant difference on mobile.

This is yet another reason why one should get root access. If ones device is not rooted, one does not fully own it. But some users realize it too late. Due to data protection, rooting requires a factory reset.

Recommended article: The sad state of personal data and infrastructure

Internal storage is not an archive.

Some people have the bad habit of treating the internal storage of their electronic devices, not limited to mobile phones, as a long-term archive.

If you are one of those people, I have bad news for you: Your smartphone's internal storage is one of the worst places to treat as long-term storage. It is among the most vulnerable locations to store data, given that it has so many points of failure.

Factors that could lead to the loss of data include: Your device could get lost, stolen, it could stop working due to a bogus update or an expired battery you can not easily replace, you might drop it on the ground or into water and the water protection fails, the USB port of a smartphone might break. These are also reasons why MicroSD is not and will never be obsolete.

In 2022, this person learnt the hard way what happens if you store data only on one portable device:

(Image source: davdreamer, Imgur )

Actual long-term storage

There is only one way to reliably store data very long term. What is it? You might not like the answer.

It's optical media. The thing you assumed to be obsolete. The type of storage media you thought is "so 2005". In particular, it is archival-grade optical media.

Optical discs (CD, DVD, Blu-ray) can not fail without warning, are water-resistant, and have an early error detection system. A hard drive is defective the second it is dropped into water, where as optical discs can survive in water for a long time. Perhaps not for years, but easily long enough for you to take them out of the water.

Optical discs are also not sensitive to electromagnetic impulses, and a disc from a defective drive can be inserted into a new and working drive.

The error detection can be done using tools such as the quality checking feature of Nero DiscSpeed and QPxTool and Opti Drive Control. However, not all vendors of optical drives support it. One that does is LITE-ON.

For the one-time inconvenience of writing data to optical discs (using K3b, ImgBurn, CDBurnerXP, InfraRecorder, or UDF packet writing), you get several decades of reliable archival.

Let him explain the rest:

(Video URL if embedding does not load.)

[I hereby release this text under the Creative Commons Attribution ShareAlike 4.0 license.]

Renate · Mar 2, 2024

I always say that I treat my portable devices as ephemeral.
I really don't care if a truck runs over one of my portable devices.
(That's only if at the time the device wasn't in my pocket.)

TheMystic · Mar 2, 2024

Hendrix7 said:
Imagine that the camera software on your smartphone did not store your pictures and video recordings in DCIM/Camera but instead into a private, locked-in directory such as /data/… , which normally is inaccessible from the outside.

This means that no other application can view the photos and videos you have taken, and you can not move them to a computer or USB stick or hard drive or any other external device.

Imagine you could only view your photographs and play back your videos through the stock gallery app that came with the smartphone. If you ran out of space, you could not move your pictures and videos out. Once you fill up the 128 GB (or however much) of internal storage that your phone comes with, that's it. You would have to delete older photos to make space.

I haven't seen this kind of a limitation on any phone, so why imagine something like this?

Hendrix7 said:
Imagine you could not back up your data anywhere else. When your smartphone breaks or stops working for whichever reason, there go your memories.

There are plenty of ways to backup data, even Automated solutions. So why imagine this too?

Hendrix7 said:
We have accepted data lock-in on mobile web browsers and messengers.

Have you noticed that none of the major mobile web browsers offer a built-in way to export your bookmarks and history and session (list of open tabs) to a file?

We haven't really accepted this, just that we were not provided any options (by popular mobile browsers). But browsers do offer syncing feature if you sign in, and that includes open tabs too. So one can always export them from the Desktop counterparts.

Hendrix7 said:
The same applies to messaging apps. Do the pre-installed messaging apps of Google and Samsung let you save your messages into a text file? No. The only way to get your messages out is to copy them individually. If you have over 9000 messages, good luck with that.

There are apps like SMS Backup & Restore that allow to do just that. Works completely offline too (if you want). You can even view the messages on a Desktop by going into the developer's website (developer claims they can't read your messages, but the privacy concern still exists).

Hendrix7 said:
Actual long-term storage

There is only one way to reliably store data very long term. What is it? You might not like the answer.

It's optical media. The thing you assumed to be obsolete. The type of storage media you thought is "so 2005". In particular, it is archival-grade optical media.

With the amount of data that we have today, one would require too many disks to store them, making it impractical. That's where cloud services come in with several layers of redundancy built-in, plus the convenience of accessing files from any device, any time, anywhere. This is not possible with optical disks or microSD cards.

Renate said:
That's only if at the time the device wasn't in my pocket.

Going1438 · Mar 4, 2024

TheMystic said:
With the amount of data that we have today, one would require too many disks to store them, making it impractical. That's where cloud services come in with several layers of redundancy built-in, plus the convenience of accessing files from any device, any time, anywhere. This is not possible with optical disks or microSD cards.

The problem of cloud is that your files are open with no encryption (not counting https) unless you bother to do an encrypting setup... Also network speed could be an issue if you're on mobile networks...

Optical discs also isn't as common as before afaik (how common do you see BD-Rs on sale?, DVD types are more common, but gonna need plenty of these to do a full backup). Solid state you'll run into issues of "data shredding" if you don't power up for some time. I'd probably recommend HDDs as the way to backup locally, with encrypted cloud backups. (And ofc do your backup strategies right, optimally 3-2-1 level backups).

Hendrix7 · Mar 6, 2024

TheMystic said:
That's where cloud services come in with several layers of redundancy built-in, plus the convenience of accessing files from any device, any time, anywhere.

Which service do you use? And do you have a paid subscription?

blackhawk · Mar 6, 2024

Going1438 said:
Solid state you'll run into issues of "data shredding" if you don't power up for some time. I'd probably recommend HDDs as the way to backup locally, with encrypted cloud backups. (And ofc do your backup strategies right, optimally 3-2-1 level backups).

There's no "data shredding" with flash memory; the cell either retains the charge or not. How many cycles it has on it also matters but at 10+k cycles it will likely be retired before its retention is compromised by leakage.
Powering it up has no effect* on stored values; you need to rewrite it to renew the cell charges. Should be good for at least 10 years. Protect from ESD exposure at all times. They can be damaged from near lightning strikes by magnetic or capacitive induction too; best kept in an earth grounded metal box ie Faraday's cage.

Memory retention diminishes with higher temperatures. With magnetic platters the same is true. Store both at 72F or so in a dry place. Never attempt to spin up a cold hdd, allow to come to room temperature first. Protect from all strong magnetic fields, ESD and near lightning strikes.

*elevated temperature however does effect retention, leave it unpowered if possible.

Renate · Mar 6, 2024

blackhawk said:
Never attempt to spin up a cold hdd, allow to come to room temperature first.

I remember years ago trying to spin up an HDD when it was below freezing.
I never had any data problems but it would need to warm up above freezing before it even whirred.
What's the exact issue? Do the heads not float?

Hendrix7 · Mar 6, 2024

Going1438 said:
The problem of cloud is that your files are open with no encryption (not counting https) unless you bother to do an encrypting setup... Also network speed could be an issue if you're on mobile networks...

Indeed. While cloud services can't get stolen or drop on the ground and break, they have different vulnerabilities.

Cloud storage providers might shut down, might terminate accounts (false positive), and have a tendency to purge inactive accounts (Reddit post). For example, how likely is OneDrive to have your files by the year 2050? Not likely.

Cloud storage providers have inactivity policies which purge long unused accounts to release disk space to new users. For example, one year for Dropbox and two years for Microsoft. Google has no formal inactivity policy, but they occasionally purge long inactive accounts like they did in December 2023. But DVDs and Blu-ray discs can not magically erase themselves. They are written once and are then zero-maintenance for decades.

In addition, a cloud service requires perpetual payment for anything more than the base storage of a few gigabytes, where as a Blu-ray disc is paid once for and stores data for decades. Admittedly, Google's free 15 GB are quite generous compared to the other cloud storage providers.

Cloud storage also comes with a privacy concern. Staff members at the cloud provider might be able to look into your data. Doesn't mean they will, but they might be technically able to. Artificial intelligence robots certainly look at your stuff.

There also is a risk of hacked accounts, but with a strong password, that is unlikely. However, write-once optical discs are inviolable to hacking.

Cloud service only is an addition to local data storage, not a replacement.

Renate said:
What's the exact issue? Do the heads not float?

I think condensation. Also, temperature shocks, meaning rapid temperature changes, are unhealthy for mechanical devices due to the parts slightly expanding.

Renate said:
I never had any data problems but it would need to warm up above freezing before it even whirred.

Perhaps temperature sensors detected the freezing temperature and prevented operation.

blackhawk · Mar 6, 2024

Do not attempt to spin up a hdd until it reaches room temperature. There's no protection for this built in. You could burn out the drive motor. Do not allow to exceed 120°F when in operation, active cooling is best for hard running drives.
NEVER drop, bump, knock, shock, etc a spinning hdd. Protect for excessive vibrations when running as well. Handle spun down ones like eggs!
Keep away from speakers (big magnets) and CRT's (degaussing coil).
Always store in a cool, dry place.
Enterprise class hdds are built for long life and data storage.

Never use a RAID 0 array for critical data storage.
Do Not encrypt or password protect data backup drives as you are the one most likely to be locked out!
Never clone or compress a data drive with media files on it as it can remove needed null marks.
Copy/paste always not move/paste!
Always verify file/folder count and for readability after copying.
Use cross platforming ie Android to PC, Mac to help reduce the chances of active malware getting into the backup database.
Vet all files before the go into the database.
Data files, foreign jpegs especially should always be in folders for further isolation. Avoid huge file folders, break large folders into smaller subfolders.

Keep at least 3 copies of critical data that are physically and electronically isolated from each other and the PC. You can never have too many backups. Time staggering to some degree is best.

Never connect a backup drive to a PC or device that is suspected of being compromised by malware. Do a full zero fill of any compromised drive if in doubt. OS loads can be replace critical data can not.
>Think< before you do things especially with backup data drives.
Labeling drives helps.

Renate · Mar 6, 2024

I presume that by this point only NAS servers should be having HDDs.
I haven't had an HDD in a while.

I still have two 250G SATA SSDs that I should reuse as offline backups.

blackhawk · Mar 6, 2024

For bookmarks and notes I've been using ColorNote. I don't trust or use browser backup at all now. Boned too many times since 2006!
ColorNote has export capabilities but you need to make sure it's auto updating to the SD card, etc.
Manually verify and/or backup the most recent data copy. Lol, had I not got my one N10+ back online I would've list a year's worth of bookmarks, etc

My bad for trusting it was still auto backing up! Still don't know what caused it to stop... lol, I'm sure there's a reason.
It's still a favorite though... just need to keep an eye on it! You can open hyperlinks directly from it.

Older version, works on Pie and Q. Glitch opening multiple hyperlinks without closing browser and ColorNote. Not a big deal but annoying. Other than this it's doing well.

Hendrix7 · Mar 6, 2024

blackhawk said:
Do not attempt to spin up a hdd until it reaches room temperature. There's no protection for this built in. You could burn out the drive motor.

If it is too cold, I don't think it will "burn" out, but it could be damaged from water condensation.

It seems Toshiba drives do have temperature detection. Which brand does not have it?

blackhawk said:
Keep away from speakers (big magnets)

Indeed. Not only because of the magnets. Subwoofers also cause vibrations that are unhealthy for hard drives.

blackhawk said:
Never use a RAID 0 array for critical data storage.

You can say that again.

EposVox unfortunately learnt the hard way:

And for SD cards, do not use cheap SD-to-USB adapters or they might interrupt the connection unexpectedly and result in a corrupted file system. Speaking from personal experience. I first thought the SD card had failed but the adapter corrupted its file system ("USBC" in boot sector).

blackhawk said:
Do Not encrypt or password protect data backup drives as you are the one most likely to be locked out!

And avoid proprietary formats for the same reason. In the long future, proprietary formats might be more difficult to access. The same already happened to Windows Write (.WRI) files and Microsoft Access databases (.MDB).

Add this to the list: Do not manage partitions before doing a full-disk image backup. This unlucky person learnt the hard way.

blackhawk said:
There's no "data shredding" with flash memory; the cell either retains the charge or not.

And cheaper flash storage is worse at retaining charge and has worse error correction.

In fact, a memory card by a no-name vendor or low-end vendor (HAMA) might corrupt your files within the span of one year. Such flash storage is at most useful for very short-term uses such as file transfers between computers.

But a memory card by a reputable brand (SanDisk, Toshiba, Transcend) can very well keep your data for a decade or more.

Higher quality flash storage has stronger error correction. The charge of the flash memory cells is usually refreshed automatically while the flash storage is supplied with power, and when data is read.

blackhawk said:
Avoid huge file folders, break large folders into smaller subfolders.

Newer file systems such as exFAT and NTFS and ext4 are better at handling large directory listings. FAT32 is unfortunately slow at handling many files in a single directory. On FAT32, I recommend staying below 10000 files per directory.

Some people also make the mistake of putting their files into split compressed archives. (.7z.001, etc.) - this should only be done where absolutely unavoidable, meaning for individual files that are too large to fit on one disc.

But for every other file, this is unnecessary and just adds the inconvenience that the files have to be unpacked before being viewable, and also one missing disc could cause all parts of the archive after that to be unreadable.

For example, if example.7z.005 is missing, example.7z.006 might be unreadable despite not being lost.

Another cause of data loss is exhausted space. A full disk could lead to failed writes. A text document might be saved as a zero-byte file.

TheMystic · Mar 6, 2024

Going1438 said:
The problem of cloud is that your files are open with no encryption (not counting https) unless you bother to do an encrypting setup... Also network speed could be an issue if you're on mobile networks...

Optical discs also isn't as common as before afaik (how common do you see BD-Rs on sale?, DVD types are more common, but gonna need plenty of these to do a full backup). Solid state you'll run into issues of "data shredding" if you don't power up for some time. I'd probably recommend HDDs as the way to backup locally, with encrypted cloud backups. (And ofc do your backup strategies right, optimally 3-2-1 level backups).

Privacy and Security are extremely important, but there is another very important factor too to consider: Convenience!

The moment one uses tech products, there is always an element of compromise with Privacy, no matter how much one tries to keep things private.

I use the Vault feature of OneDrive (it is encrypted) to store all my documents and other very important files. For everything else, such as camera roll, I use multiple cloud services (from different service providers) to back them up, in addition to 2 local backups.

Since I use multiple devices, it is extremely convenient for me to access pretty much all my files from any device, anytime, anywhere. This is not possible if I were to stick with microSD cards, optical disks or HDDs!

Hendrix7 said:
Which service do you use? And do you have a paid subscription?

Multiple services, but primarily Microsoft 365 (Family plan) and Google One.

TheMystic · Mar 6, 2024

More than 10 years back, I was playing around with Windows using command line, and by mistake used the Diskpart command from recovery, without fully understanding what it was going to do. When my system booted, I found a clean disk: everything, including precious memories, were gone! After spending couple of sleepless nights and trying out several tools, I finally found one that actually recovered almost 100% of my files. It was nothing but God's grace that got me back my files. I learnt my lesson very well that day, and set up a backup plan that has since worked flawlessly for me.

V0latyle · Mar 6, 2024

Nothing particularly revolutionary here....

Anyone who values their data enough to ensure it's accessible knows the value of redundant backups. Storage has been getting progressively cheaper for years, and if you don't want to spend the money on a NAS with a RAID array, just outsource it - there's plenty of cloud storage options with integral encryption.

Privacy issues aside, Google has been largely pioneering this with Google Photos and Google One for the Pixels, although unlimited Photos backup is no longer available for the 5a and newer.

Hendrix7 · Mar 6, 2024

blackhawk said:
For bookmarks and notes I've been using ColorNote. I don't trust or use browser backup at all now. Boned too many times since 2006!

If any online content is valuable, download it. Bookmarking is not enough.

A YouTube video that exists today is not guaranteed to exist tomorrow.

blackhawk · Mar 6, 2024

Hendrix7 said:
If it is too cold, I don't think it will "burn" out, but it could be damaged from water condensation.

It seems Toshiba drives do have temperature detection. Which brand does not have it?

Never spin up a cold drive... water condensation isn't the issue. Cold bearings and metal contraction are. Tolerances are extremely tight. Temperature detection doesn't mean it won't spin up at low temperatures...

Hendrix7 said:
Indeed. Not only because of the magnets. Subwoofers also cause vibrations that are unhealthy for hard drives.

You can say that again.

EposVox unfortunately learnt the hard way:

And for SD cards, do not use cheap SD-to-USB adapters or they might interrupt the connection unexpectedly and result in a corrupted file system. Speaking from personal experience. I first thought the SD card had failed but the adapter corrupted its file system ("USBC" in boot sector).

And avoid proprietary formats for the same reason. In the long future, proprietary formats might be more difficult to access. The same already happened to Windows Write (.WRI) files and Microsoft Access databases (.MDB).

Add this to the list: Do not manage partitions before doing a full-disk image backup. This unlucky person learnt the hard way.

And cheaper flash storage is worse at retaining charge and has worse error correction.

In fact, a memory card by a no-name vendor or low-end vendor (HAMA) might corrupt your files within the span of one year. Such flash storage is at most useful for very short-term uses such as file transfers between computers.

But a memory card by a reputable brand (SanDisk, Toshiba, Transcend) can very well keep your data for a decade or more.

A flash card that doesn't operate perfectly in a single device after formatting in that device should be discarded. Do Not "share" memory cards between devices. Upload/download through the Android to the card. I can get away with sharing a card between my N10+'s but I don't recommend as it can cause the card to go corrupt. Like sharing between two cameras without formatting first.

Yes always use a name brand card. Get V-30 rated or higher.

Hendrix7 said:
Higher quality flash storage has stronger error correction. The charge of the flash memory cells is usually refreshed automatically while the flash storage is supplied with power, and when data is read.

Don't count on it refreshing cell charges. Dead cells are mapped out though.

Hendrix7 said:
Newer file systems such as exFAT and NTFS and ext4 are better at handling large directory listings. FAT32 is unfortunately slow at handling many files in a single directory. On FAT32, I recommend staying below 10000 files per directory.

Some people also make the mistake of putting their files into split compressed archives. (.7z.001, etc.) - this should only be done where absolutely unavoidable, meaning for individual files that are too large to fit on one disc.

But for every other file, this is unnecessary and just adds the inconvenience that the files have to be unpacked before being viewable, and also one missing disc could cause all parts of the archive after that to be unreadable.

For example, if example.7z.005 is missing, example.7z.006 might be unreadable despite not being lost.

Another cause of data loss is exhausted space. A full disk could lead to failed writes. A text document might be saved as a zero-byte file.

NTFS always for Windows.
Always leave 10% headroom, more for OS drives.
The reason for the folders is they act to isolate scripted jpegs or other infected files. These may not be detected by antivirus and can damage any files in that folder! I've seen both Window and Android versions of these, nothing detected them at that time.
I do the same with mp4's etc. Each movie or song album has its own folder. Helps with organizing as well.

All downloads stay in the download folder until vetted, open all jpegs and media files, see they work and look for any changes in that folder afterwards. Any unknown downloads delete without opening. Use folders in the download folder ie music, apps, movies, etc. Again to protect and organize them.

☠Do not clone (lose needed null marks), convert or compress wav (lose of HDCD subtext) files!☠

blackhawk · Mar 6, 2024

Hendrix7 said:
If any online content is valuable, download it. Bookmarking is not enough.

A YouTube video that exists today is not guaranteed to exist tomorrow.

Depends on it's value to you. Many times it's the site itself that's of interest.
Youtube quality generally sucks. I try for high res files whenever possible especially for music. Flac or wav for music.
If it's something I like on internet archive I grab it though!

blackhawk · Mar 6, 2024

V0latyle said:
Nothing particularly revolutionary here....

Anyone who values their data enough to ensure it's accessible knows the value of redundant backups. Storage has been getting progressively cheaper for years, and if you don't want to spend the money on a NAS with a RAID array, just outsource it - there's plenty of cloud storage options with integral encryption.

Privacy issues aside, Google has been largely pioneering this with Google Photos and Google One for the Pixels, although unlimited Photos backup is no longer available for the 5a and newer.

I won't let Gookill touch my images. It has failed me miserably in multiple ways in the past. I can only trust Google to fail me again... like a bad gf.

Hendrix7 · Mar 6, 2024

blackhawk said:
☠Do not clone (lose needed null marks), convert or compress wav (lose of HDCD subtext) files!☠

Compressing using file archivers (ZIP, 7z, GZIP) is not going to change the content. And cloning copies the exact same bytes from the source drive to the target drive.

Audio CDs don't work using conventional file systems but tracks, so one has to use software that can read all the data one wants backed up.

blackhawk said:
I won't let Gookill touch my images. It has failed me miserably in multiple ways in the past. I can only trust Google to fail me again... like a bad gf.

Hasn't failed me so far, but I would still not trust cloud services with any irreplaceable data I don't have stored locally. Cloud services are a convenient addition, not a replacement for local storage.

Renate said:
I still have two 250G SATA SSDs that I should reuse as offline backups.

For a local backup, I recommend choosing a different model of hard drive or SSD than the source drive, so any manufacturing errors do not affect both the source and the backup, see the Seagate DM001 disaster.

DM001 was nicknamed "doom" and "doom1" on various computer forums.

The massive problem of data lock-in

Member

We have accepted data lock-in on mobile web browsers and messengers.​

Internal storage is not an archive.​

Actual long-term storage​

Recognized Contributor / Inactive Recognized Dev

Senior Member

We have accepted data lock-in on mobile web browsers and messengers.​

Actual long-term storage​

Senior Member

Member

Senior Member

Recognized Contributor / Inactive Recognized Dev

Member

Senior Member

Recognized Contributor / Inactive Recognized Dev

Senior Member

Attachments

Member

Senior Member

Senior Member

Forum Moderator

Member

Senior Member

Senior Member

Senior Member

Member

Similar threads

Top Liked Posts

We have accepted data lock-in on mobile web browsers and messengers.​

Internal storage is not an archive.​

Actual long-term storage​

We have accepted data lock-in on mobile web browsers and messengers.

Internal storage is not an archive.

Actual long-term storage

We have accepted data lock-in on mobile web browsers and messengers.

Actual long-term storage

We have accepted data lock-in on mobile web browsers and messengers.

Internal storage is not an archive.

Actual long-term storage