Wikiversity - User contributions [en]

Firefox

2025-02-14T13:00:52Z

Elominius: /* Advanced configuration */ forceMediaMemoryCache

==Setup==
* [http://support.mozilla.com/en-US/kb/How+to+make+Firefox+the+default+browser How to make Firefox the default browser]

==Shortcuts==
* CTRL+W closes a tab
* CTRL+T opens a new tab
* CTRL+SHIFT+T reopens the last closed tab
* CTRL+↹Tabulator; CTRL+Shift+↹Tabulator; CTRL+PgUp/PgDn: switch through tabs.
* CTRL+Shift+PgUp/PgDn: move opened tab left or right respectively.
* F6 or Ctrl+L : Gets you right up into the Address/URL bar.
* F5 : Reload the page.
* CTRL+F5: Clear cache for the current page and reload the page.
* / or Ctrl+F: Search/find text on page
* Ctrl+K : Takes you to the Firefox search box.
* Ctrl+U : View the page’s source code.
* F11 : View the page in full-screen mode.
* Ctrl+W : Closes the active tab.
* Ctrl+<kbd>=</kbd> : Resets font size.
* Ctrl+<kbd>+</kbd> : Increases font size.
* Ctrl+<kbd>-</kbd> : Decreases font size.
* Ctrl+S : Save page
* Ctrl+Shift+I, F12: Open web development tools.

== Downloading ==
There are four main ways to download an internet resource:

* <kbd>Ctrl+S</kbd> – Download currently viewed page.

In the file saving dialogue, "Web Page, complete" takes the loaded page from memory and stores resources such as multimedia and style sheets and JavaScripts as individual files inside a <code>_files</code> suffix directory. "Web Page, HTML only" saves the verbatim page source code downloaded from the server.

* "Save Link as" from right-click context menu (saves verbatim page source code)
* Dragging the URL or tab into the Download arrow in the tool bar.
* Pasting an URL from the clipboard into the opened download history (in the "Library" window or <code>about:downloads</code>).

Additionally, the page source code can be saved by copying it from the web development tools and pasted into a text editor.

== Profiles ==
User profiles allow separating configuration, extensions, bookmarks, and history within Firefox. By default, there is one profile. The profile manager allows creating additional profiles. It can be opened by launching the Firefox process through a command line with the <code>-profilemanager</code> parameter, without spaces.

Profile folders are typically located under <code>%AppData%\Mozilla\Firefox\</code> on Microsoft Windows,<code>~/Library/Application Support/Firefox/</code> on MacOS, and <code>~/.mozilla/firefox/</code> on Linux and BSD. The profile manager allows manually specifying custom profile directories.<ref>[https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles?redirectslug=profile-manager-create-and-remove-firefox-profiles&redirectlocale=en-US ''Profile Manager - Create, remove or switch Firefox profiles'' – Mozilla Support (knowledge base)]</ref>

Relevant files in the profile folder are <code>places.sqlite</code> for browsing history and download history and bookmarks, <code>cookies.sqlite</code>, <code>prefs.js</code> for user preferences, and <code>containers.json</code> for configuration of multi-account containers. Sessions are stored in <code>sessionstore.jsonlz4</code> when closing the browser (before Firefox 56 of 2017, it was stored as plain JSON into <code>sessionstore.json</code>), and backed up to <code>sessionstore-backups/recovery.jsonlz4</code> and <code>sessionstore-backups/recovery.baklz4</code> while browsing for recovery in case of an unexpected process termination. The previous session is backed up to <code>sessionstore-backups/previous.jsonlz4</code>. <ref>[https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data Profiles - Where Firefox stores your bookmarks, passwords and other user data – Mozilla Support]</ref>

== Advanced configuration ==
On the desktop edition (and earlier mobile versions), the <code>about:config</code> page allows fine-tuning the browser. The search bar at the top facilitates finding properties.

Notable properties are:

* <code>browser.urlbar.trimURLs</code> – may hide the protocol from the URL bar. Deactivate to always show the full URL including the protocol. Activated by default.
* <code>browser.tabs.insertAfterCurrent</code> – open new tabs next to the currently active tab instead of at the end of the tab list.
* <code>javascript.enabled</code> – deactivating usually increases speed and decreases CPU usage of [[:w:Progressive enhancement|progressively enhancing]] sites, but features implemented using JavaScript such as [[:mw:Extension:WikiEditor|the toolbar of the wikitext form]] and JavaScript-based sites such as Twitter web app will not work. Activated by default.
* <code>browser.backspace_action</code> – this parameter adjusts the function of the backspace key if no text input field is active. <code>0</code>: navigate to last page, <code>1</code>: scroll up, if ''Shift'' key is held then scroll down. <code>2</code>: deactivate.
* <code>dom.event.contextmenu.enabled</code> – allows JavaScript to interfere with right-click context menu. Deactivating might interfere with some sites' functionality. Activated by default.
* <code>dom.event.clipboardevents.enabled</code> – allows JavaScript to evaluate on-page text selection. Deactivating might interfere with some sites' functionality. Activated by default.
* <code>browser.download.autohideButton</code> – hide download button if no downloads in current session. Activated by default.
* <code>browser.download.alwaysOpenPanel</code> – Open list of recent downloads after a download finished.
* <code>browser.download.lastDir</code>, <code>browser.open.lastDir</code> – last directory to which a file was downloaded or from which a file was opened through the file picker dialogue.
* <code>browser.startup.homepage</code> – page displayed when starting a new session (if not restoring the previous session), and when pressing the button with the house icon.
* <code>dom.ipc.processCount.web</code> – number of processes across which web content is distributed. Higher count increases performance but consumes more memory, thus recommended for computers with much RAM. Four by default.
* <code>general.autoScroll</code> – enable scrolling using middle mouse button press.
* <code>devtools.chrome.enabled</code> – enable write access to browser console, accessed using Ctrl+Shift+J.
* <code>browser.cache.disk.capacity</code>, <code>browser.cache.offline.capacity</code> disk cache capacity, mainly used for static resources to save bandwidth. Difference between values unclear yet. – Values in Kilobytes.
* <code>places.history.expiration.transient_current_max_pages</code>: number of entries retained in the browsing history database (<code>places.sqlite</code> file). According to a moderator of Mozilla's support forum, the value is read-only by default but can be adjusted through the <code>places.history.expiration.max_pages</code> property, which has to be added manually.<ref>[https://support.mozilla.org/en-US/questions/1275209#answer-1274292 Answer to ''FF 71. Cannot change value in setting places.history.expiration.transient_current_max_pages'' | Firefox Support Forum | Mozilla Support (December 17, 2019)]</ref>
* <code>accessibility.blockautorefresh</code> – blocks "meta refreshes" and "meta redirects", meaning refreshes and redirects using the <code>meta http-equiv</code> tag, as well as redirects instructed by the <code>refresh</code> [[:w:HTTP header|HTTP header]]. However, it does not block JavaScript-based redirects and refreshes, i. e. <code>document.location.href</code>.<ref>[https://support.mozilla.org/en-US/questions/1327430 ''Enable/disable automatic redirect. | Firefox Support Forum | Mozilla Support'' (2/26/21)]</ref>
* <code>security.csp.enable</code> – Claimed by Mozilla to increase security against "cross-site scripting" (XSS) attacks.<ref>[https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP Content Security Policy (CSP) - HTTP] from Mozilla Developers Network</ref> User reports suggest it interferes with the function of [[:w:JavaScript bookmarklet|bookmarklet]]s.<ref>[https://superuser.com/questions/586063/how-to-disable-csp-in-firefox-for-just-bookmarklets ''How to disable CSP in Firefox for just bookmarklets?'' – Super User]</ref>
* <code>permissions.default.image</code> – 1: Load images as usual (default); 2: Do not load images (usually for testing purposes or to save bandwidth); 3: Only load images from the same domain.<ref>https://support.mozilla.org/en-US/questions/981640</ref>
* <code>media.videocontrols.picture-in-picture.video-toggle.enabled</code> (or similar): show the picture-in-picture button in video players.
* <code>security.dialog_enable_delay</code> – delay in milliseconds until the confirmation button in some dialogue boxes like downloading an executable file or installing an extension gets unlocked. Until then, it is [[:w:Grayed_out|"grayed out"]]. Default value: 1000 (1 second).<ref>[http://kb.mozillazine.org/Disable_extension_install_delay_-_Firefox ''Disable extension install delay - Firefox - MozillaZine Knowledge Base'']</ref>
* <code>browser.cache.disk.enable</code> – enable saving sites' resources locally to speed up loading.
* <code>browser.cache.disk.capacity</code> – disk cache size in kilobytes.
* <code>browser.urlbar.update2.engineAliasRefresh</code> – allow adding custom search engines.<ref>[https://superuser.com/questions/7327/how-to-add-a-custom-search-engine-to-firefox/1756774#1756774 How to add a custom search engine to Firefox? - Super User]</ref>
* <code>accessibility.typeaheadfind.matchesCountLimit</code> – Sets the limit for counting text matches when searching text on the current page. A high limit can slow down the browser on long pages. The default is 1000.
* <code>browser.privatebrowsing.forceMediaMemoryCache</code> – While in [[:w:private_browsing|private browsing]], video and audio content is only cached into the computer's RAM, not the data storage. No such option exists for normal browsing as of early 2025.

== System pages ==
* <code>about:memory</code> – internal process viewer.
* <code>about:processes</code> – newer internal process manager. Introduced with Firefox 67 in early 2019.<ref>https://www.ghacks.net/2019/03/01/firefox-67-automatically-unload-unused-tabs-to-improve-memory/</ref>
* <code>about:performance</code> – similar to <code>about:processes</code>, but with more beginner-friendly worded user interface, such as "Energy impact" instead of "CPU". Introduced in late 2018.<ref>[https://www.ghacks.net/2018/10/11/this-is-firefoxs-upcoming-aboutperformance-page-huge-improvements/ This is Firefox's upcoming about:performance page (huge improvements) – Martin Brinkmann, 2018-10-11]</ref>

==See also==
* [[w:Iceweasel|Iceweasel]] browser
* [[devmo:|Firefox Development Wiki]]
* [[MozillaZineKB:|Mozilla Knowledge-Base Wiki]]
* [[links (browser)]], [[lynx (browser)]], [[Elinks (browser)]] command line browsers
* [[CURL (software)|cURL]]

==External links==
* [http://www.qedoc.org/en/index.php?title=Mozilla_Firefox Mozilla Firefox] - Interactive learning quiz at Qedoc.org

[[Category:Firefox]]
[[Category:Web browsers]]

Firefox

2025-02-14T12:56:24Z

Elominius: /* Advanced configuration */ accessibility.typeaheadfind.matchesCountLimit - I'm surprised I never heard of this before. Hopefully useful to others too.

==Setup==
* [http://support.mozilla.com/en-US/kb/How+to+make+Firefox+the+default+browser How to make Firefox the default browser]

==Shortcuts==
* CTRL+W closes a tab
* CTRL+T opens a new tab
* CTRL+SHIFT+T reopens the last closed tab
* CTRL+↹Tabulator; CTRL+Shift+↹Tabulator; CTRL+PgUp/PgDn: switch through tabs.
* CTRL+Shift+PgUp/PgDn: move opened tab left or right respectively.
* F6 or Ctrl+L : Gets you right up into the Address/URL bar.
* F5 : Reload the page.
* CTRL+F5: Clear cache for the current page and reload the page.
* / or Ctrl+F: Search/find text on page
* Ctrl+K : Takes you to the Firefox search box.
* Ctrl+U : View the page’s source code.
* F11 : View the page in full-screen mode.
* Ctrl+W : Closes the active tab.
* Ctrl+<kbd>=</kbd> : Resets font size.
* Ctrl+<kbd>+</kbd> : Increases font size.
* Ctrl+<kbd>-</kbd> : Decreases font size.
* Ctrl+S : Save page
* Ctrl+Shift+I, F12: Open web development tools.

== Downloading ==
There are four main ways to download an internet resource:

* <kbd>Ctrl+S</kbd> – Download currently viewed page.

In the file saving dialogue, "Web Page, complete" takes the loaded page from memory and stores resources such as multimedia and style sheets and JavaScripts as individual files inside a <code>_files</code> suffix directory. "Web Page, HTML only" saves the verbatim page source code downloaded from the server.

* "Save Link as" from right-click context menu (saves verbatim page source code)
* Dragging the URL or tab into the Download arrow in the tool bar.
* Pasting an URL from the clipboard into the opened download history (in the "Library" window or <code>about:downloads</code>).

Additionally, the page source code can be saved by copying it from the web development tools and pasted into a text editor.

== Profiles ==
User profiles allow separating configuration, extensions, bookmarks, and history within Firefox. By default, there is one profile. The profile manager allows creating additional profiles. It can be opened by launching the Firefox process through a command line with the <code>-profilemanager</code> parameter, without spaces.

Profile folders are typically located under <code>%AppData%\Mozilla\Firefox\</code> on Microsoft Windows,<code>~/Library/Application Support/Firefox/</code> on MacOS, and <code>~/.mozilla/firefox/</code> on Linux and BSD. The profile manager allows manually specifying custom profile directories.<ref>[https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles?redirectslug=profile-manager-create-and-remove-firefox-profiles&redirectlocale=en-US ''Profile Manager - Create, remove or switch Firefox profiles'' – Mozilla Support (knowledge base)]</ref>

Relevant files in the profile folder are <code>places.sqlite</code> for browsing history and download history and bookmarks, <code>cookies.sqlite</code>, <code>prefs.js</code> for user preferences, and <code>containers.json</code> for configuration of multi-account containers. Sessions are stored in <code>sessionstore.jsonlz4</code> when closing the browser (before Firefox 56 of 2017, it was stored as plain JSON into <code>sessionstore.json</code>), and backed up to <code>sessionstore-backups/recovery.jsonlz4</code> and <code>sessionstore-backups/recovery.baklz4</code> while browsing for recovery in case of an unexpected process termination. The previous session is backed up to <code>sessionstore-backups/previous.jsonlz4</code>. <ref>[https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data Profiles - Where Firefox stores your bookmarks, passwords and other user data – Mozilla Support]</ref>

== Advanced configuration ==
On the desktop edition (and earlier mobile versions), the <code>about:config</code> page allows fine-tuning the browser. The search bar at the top facilitates finding properties.

Notable properties are:

* <code>browser.urlbar.trimURLs</code> – may hide the protocol from the URL bar. Deactivate to always show the full URL including the protocol. Activated by default.
* <code>browser.tabs.insertAfterCurrent</code> – open new tabs next to the currently active tab instead of at the end of the tab list.
* <code>javascript.enabled</code> – deactivating usually increases speed and decreases CPU usage of [[:w:Progressive enhancement|progressively enhancing]] sites, but features implemented using JavaScript such as [[:mw:Extension:WikiEditor|the toolbar of the wikitext form]] and JavaScript-based sites such as Twitter web app will not work. Activated by default.
* <code>browser.backspace_action</code> – this parameter adjusts the function of the backspace key if no text input field is active. <code>0</code>: navigate to last page, <code>1</code>: scroll up, if ''Shift'' key is held then scroll down. <code>2</code>: deactivate.
* <code>dom.event.contextmenu.enabled</code> – allows JavaScript to interfere with right-click context menu. Deactivating might interfere with some sites' functionality. Activated by default.
* <code>dom.event.clipboardevents.enabled</code> – allows JavaScript to evaluate on-page text selection. Deactivating might interfere with some sites' functionality. Activated by default.
* <code>browser.download.autohideButton</code> – hide download button if no downloads in current session. Activated by default.
* <code>browser.download.alwaysOpenPanel</code> – Open list of recent downloads after a download finished.
* <code>browser.download.lastDir</code>, <code>browser.open.lastDir</code> – last directory to which a file was downloaded or from which a file was opened through the file picker dialogue.
* <code>browser.startup.homepage</code> – page displayed when starting a new session (if not restoring the previous session), and when pressing the button with the house icon.
* <code>dom.ipc.processCount.web</code> – number of processes across which web content is distributed. Higher count increases performance but consumes more memory, thus recommended for computers with much RAM. Four by default.
* <code>general.autoScroll</code> – enable scrolling using middle mouse button press.
* <code>devtools.chrome.enabled</code> – enable write access to browser console, accessed using Ctrl+Shift+J.
* <code>browser.cache.disk.capacity</code>, <code>browser.cache.offline.capacity</code> disk cache capacity, mainly used for static resources to save bandwidth. Difference between values unclear yet. – Values in Kilobytes.
* <code>places.history.expiration.transient_current_max_pages</code>: number of entries retained in the browsing history database (<code>places.sqlite</code> file). According to a moderator of Mozilla's support forum, the value is read-only by default but can be adjusted through the <code>places.history.expiration.max_pages</code> property, which has to be added manually.<ref>[https://support.mozilla.org/en-US/questions/1275209#answer-1274292 Answer to ''FF 71. Cannot change value in setting places.history.expiration.transient_current_max_pages'' | Firefox Support Forum | Mozilla Support (December 17, 2019)]</ref>
* <code>accessibility.blockautorefresh</code> – blocks "meta refreshes" and "meta redirects", meaning refreshes and redirects using the <code>meta http-equiv</code> tag, as well as redirects instructed by the <code>refresh</code> [[:w:HTTP header|HTTP header]]. However, it does not block JavaScript-based redirects and refreshes, i. e. <code>document.location.href</code>.<ref>[https://support.mozilla.org/en-US/questions/1327430 ''Enable/disable automatic redirect. | Firefox Support Forum | Mozilla Support'' (2/26/21)]</ref>
* <code>security.csp.enable</code> – Claimed by Mozilla to increase security against "cross-site scripting" (XSS) attacks.<ref>[https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP Content Security Policy (CSP) - HTTP] from Mozilla Developers Network</ref> User reports suggest it interferes with the function of [[:w:JavaScript bookmarklet|bookmarklet]]s.<ref>[https://superuser.com/questions/586063/how-to-disable-csp-in-firefox-for-just-bookmarklets ''How to disable CSP in Firefox for just bookmarklets?'' – Super User]</ref>
* <code>permissions.default.image</code> – 1: Load images as usual (default); 2: Do not load images (usually for testing purposes or to save bandwidth); 3: Only load images from the same domain.<ref>https://support.mozilla.org/en-US/questions/981640</ref>
* <code>media.videocontrols.picture-in-picture.video-toggle.enabled</code> (or similar): show the picture-in-picture button in video players.
* <code>security.dialog_enable_delay</code> – delay in milliseconds until the confirmation button in some dialogue boxes like downloading an executable file or installing an extension gets unlocked. Until then, it is [[:w:Grayed_out|"grayed out"]]. Default value: 1000 (1 second).<ref>[http://kb.mozillazine.org/Disable_extension_install_delay_-_Firefox ''Disable extension install delay - Firefox - MozillaZine Knowledge Base'']</ref>
* <code>browser.cache.disk.enable</code> – enable saving sites' resources locally to speed up loading.
* <code>browser.cache.disk.capacity</code> – disk cache size in kilobytes.
* <code>browser.urlbar.update2.engineAliasRefresh</code> – allow adding custom search engines.<ref>[https://superuser.com/questions/7327/how-to-add-a-custom-search-engine-to-firefox/1756774#1756774 How to add a custom search engine to Firefox? - Super User]</ref>
* <code>accessibility.typeaheadfind.matchesCountLimit</code> – Sets the limit for counting text matches when searching text on the current page. A high limit can slow down the browser on long pages. The default is 1000.

== System pages ==
* <code>about:memory</code> – internal process viewer.
* <code>about:processes</code> – newer internal process manager. Introduced with Firefox 67 in early 2019.<ref>https://www.ghacks.net/2019/03/01/firefox-67-automatically-unload-unused-tabs-to-improve-memory/</ref>
* <code>about:performance</code> – similar to <code>about:processes</code>, but with more beginner-friendly worded user interface, such as "Energy impact" instead of "CPU". Introduced in late 2018.<ref>[https://www.ghacks.net/2018/10/11/this-is-firefoxs-upcoming-aboutperformance-page-huge-improvements/ This is Firefox's upcoming about:performance page (huge improvements) – Martin Brinkmann, 2018-10-11]</ref>

==See also==
* [[w:Iceweasel|Iceweasel]] browser
* [[devmo:|Firefox Development Wiki]]
* [[MozillaZineKB:|Mozilla Knowledge-Base Wiki]]
* [[links (browser)]], [[lynx (browser)]], [[Elinks (browser)]] command line browsers
* [[CURL (software)|cURL]]

==External links==
* [http://www.qedoc.org/en/index.php?title=Mozilla_Firefox Mozilla Firefox] - Interactive learning quiz at Qedoc.org

[[Category:Firefox]]
[[Category:Web browsers]]

Smartphone

2025-02-14T08:09:44Z

Elominius: /* Camera */ self-timer

[[File:Smartphone icon.svg|thumb|right|Smartphone]]

A smartphone is a portable device combining functionality of many solid-state everyday life tools in one unit, including digital camera, camcorder, flashlight torch, music player, book reader, web browser, messenger, radio, environmental sensors like air pressure, and more. The functionality varies among models.

Supplementary to [[:w:Smartphone|the Wikipedia article]], this resource documents tips and guides on usage.

== Charging port care ==
To prevent unnecessary wear on the charging port, connector, and cable endings, grab the handle and not the cable. The two handles are between cable and the metal pieces that enter the port. Push or pull the connector perpendicularly to the charging port without applying sideways pressure that could wear down the connector and port faster.

Slightly and alternatingly moving the connector sideways in parallel to the USB connector's flat surface (longer edge, horizontally to the charging port) is harmless and could facilitate connecting and disconnecting, whereas downward pressure (vertically to charging port against the connector's longer edge) is unhealthy for the connector endings and should be prevented.

Inserting a more worn charging connector wears down the phone's port less, since it requires less insertion force, but has a higher likelihood of undesirably disconnecting.

Wearing down the charging port can be limited through wireless charging. While more convenient, it is slower and produces more heat than wired charging due to the lower efficiency of wireless power transmission. As such, wired charging is more suitable in a hurry.

== Battery care ==
Battery care is of increasing concern as a result of the popularization of designs that deny its replacement, typically the shortest-lived component of the unit.

Signs of an expiring phone battery are quick battery drain and system outages caused by voltage starvation.

In order to delay the battery expiry, stress to it needs to be limited by foregoing fast charging to the extent possible, as it causes heat that stresses the battery, and the operational battery capacity needs to be limited to a centered range between approximately 25% and 75%, as too high and too low charging states wear it down faster. If no automated process is able to limit the charging state, the power needs to be disconnected manually at a charging state of around 70 to 80 percent, as well as be recharged in time to prevent drain below 20%.

The smartphone can optionally be cooled using a portable external USB-powered fan. This both extends the life span of the battery and might increase charging speed if it was throttled by an overheating protection mechanism.

Possible measures to reduce power consumption include deactivating connectivity features such as WiFi and Bluetooth when unneeded, and monitoring background apps' activity. Note that some tasks such as using cellular internet and high-definition video recording inevitably consume much power. Consider bringing an external battery bank ("[[:w:power bank|power bank]]") with you.

When exposed to cold temperatures which could occur during the winter season, a smartphone might power down as a result of the cold temperatures slowing down the chemical reactions in the battery cell, causing an undervoltage condition. Smaller and older batteries are weakened more by cold temperatures. If no external power source is available, it is recommended to warm up the smartphone for a few minutes using your body.

== Physical protection ==
It is recommended to use both a protective case and a screen protector. Even cheap ones provide significant protection compared to none, and are far cheaper and easier to replace than built-in smartphone parts should damage occur.

Smartphone cases may be equipped with useful features such as a kickstand.

When removing a smartphone case that is difficult to remove, apply pressure to either of the shorter shorter edges (i.e. bottom or top side from vertical view), since they are more robust against the pressure.

== Charging speed ==
[[File:ACadapterJUP44100j5createPDCharger20210905TestUSB.jpg|thumb|USB multimeter]]
Using a cable with low electrical impedance (resistance) may speed up charging. Cables rated to support higher currents such as 5 Amp. have thicker copper wires, thus less resistance. Longer cables (e.g. 3 metres) tend to have a higher resistance, though the thickness of the copper wires plays a more important role.

Charging protocols such as [[:w:Charge_controller#Quick_Charge_and_Pump_Express|''Quick Charge'' by Qualcomm and ''Pump Express'' by MediaTek]], which use elevated voltages, are likely compatible with existing USB cables and don't mind long cables, but produce more heat in the end device due to necessary internal voltage downregulation.

Charging speed can be verified using a USB multimeter as depicted.

== Camera ==
=== Controls ===
It has been widely established across smartphone vendors that focusing is done by tapping on an object inside the viewfinder, [[:w:Exposure_compensation|compensating]] the [[:w:exposure value|exposure value]] (brightness) is done by swiping up and down, and focus and exposure are locked by holding on the viewfinder for between half a second to a second.

After tapping on the viewfinder, a rectangle or circle usually denotes the focused position, and it might turn green when focused successfully and otherwise red. The exposure value might remain manually adjustable while focus and exposure are locked. Depending on the camera software, the availability of these gestures could be restricted in video recording mode, slow motion recording mode, and for the front-facing camera.

<ref>{{cite web |title=How to Use Autofocus and Auto Exposure Lock on Your iPhone Camera |url=https://www.iphonelife.com/content/tip-day-why-and-how-use-autofocus-and-auto-exposure-lock-your-iphone-camera |website=www.iphonelife.com |access-date=15 October 2022 |language=en |date=11 August 2015}}</ref>
<ref>{{cite web |title=How to Adjust Exposure on Galaxy S7 • Android Flagship |url=https://androidflagship.com/22675-adjust-exposure-on-galaxy-s7/ |website=Android Flagship |access-date=15 October 2022 |date=6 October 2016}}</ref>

=== Orientation ===
[[File:Example_vertical_video_on_widescreen.JPG|thumb|Appearance of vertical video on wide screen]]
It is recommended to film horizontally and to avoid rotating the unit while recording.

Filming horizontally is recommended as the resulting footage will gracefully fill computer monitors' and televisions' screen surface, whereas vertical video appears as a slit.

Vertically filmed footage would need to be cropped or zoomed in to fill the entire screen, not only resulting in a lower-resolution crop of the video, but depending on video players' capability to do so during playback. Cropping in a video editor either requires time-taking rendering to a separate video, or sufficient processing power for real-time playback.

Orientation should remain constant throughout recording, as filming in a different orientation than when recording was initiated results in a misrotated picture. Counter-measures such as correcting rotation using metadata from the accelerometer or gyroscope have not been implemented as of 2021, and early future implementations might be proprietary (closed-source), thus not widely supported among video players.

To ensure photos and videos are saved in the correct orientation, first look at the user interface's elements. If their orientation does not match the desired orientation, which can happen shortly after launching or when pointing the camera nearly perpendicularly, tilt the device upwards for a short moment to correct the rotation assumed by the software.

Some camera software locks the rotation of the user interface elements while filming or warns the user about rotation.

=== Resolutions ===
The camera software typically lets one adjust the resolution for photography and filming. A higher resolution produces better-quality images and videos, but demands more battery power and storage. Adjust the resolution based on the value of a moment.

If you are low on storage or battery charge, you might want to select a lower resolution to extend the remaining recording time.

; The screen argument
<blockquote>''I don't have a 4K screen, so why should I film in 4K?''</blockquote>

Not having a screen that matches the dimensions of a video resolution option is not an argument not to record in that resolution, since one might acquire such a screen in future, and higher resolutions still deliver a higher quality due to higher [[:w:bit rate|bit rate]]s and [[:w:chroma subsampling|chroma subsampling]]. In addition, higher resolutions allow for more magnification in later [[video editing]], as well as extraction of higher-resolution still pictures. In simple words, if one wants the best possible 1080p (Full HD), one should film in 2160p (4K).

Even if a 2160p video is never watched on a 2160p screen, still frames can be extracted as standalone photographs. 2160p translates into over 8 megapixels, whereas a 1080p frame is just above 2 megapixels.

These benefits make video footage with a higher resolution more ''[[:wikt:future-proof|future-proof]]'', meaning potentially more useful in the future.

The same principle applies to frame rates. While the only purpose of recording at high framerates such as 120 frames per second used to be slowing down the video to create a slow motion effect, screens with a refresh rate of 120 are becoming commonplace on mobile phones, so a video recorded at 120 frames per second in 2015 can be viewed at real-time speed for a smooth effect on a {{#time:Y}} smartphone.

; Purity
[[File:Colorcomp.jpg|thumb|In [{{filepath:Colorcomp.jpg}} full size], this image shows the quality difference between four subsampling schemes.]]
Since the software of smartphone cameras (and other consumer-grade video cameras) encodes the footage with a chroma subsampling scheme of <code>4:2:0</code>, the colour information is stored at only at the quarter of the resolution as the brightness information, in order to compress far more efficiently with only a slight quality loss noticeable to the human eye. This means that 1080p Full HD video (1920×1080 pixels) only has 540p (960×540 pixels) of colour information, whereas 2160p 4K video (3840×2160 pixels) has 1920×1080 pixels of colour information, making it resemble pure <code>4:4:4</code> subsampling at 1080p. Due to the higher bit rate of 2160p 4K video, compression artefacts are less visible even when played back on lower-resolution screens.<ref>[https://www.youtube.com/watch?v=kIf9h2Gkm_U Why Does 4K Look Better on 1080p Monitors – The Daily Mote (May 14th, 2015)]</ref>

=== Microphones ===
Typically, there are two microphone holes, one located at the top edge and one at the bottom edge, to record spatial stereo audio while holding the smartphone horizontally for recording.

Make sure your hands do not cover the microphones of your smartphone while recording video, or the audio quality will suffer.

=== Self-timer ===
In low light, a possibility to prevent shakiness is to make use of the self-timer.

Set a short time like two or three seconds (depending on which option exists in the software) so you can use this time to change your grip and posture to as steady as possible after pushing the shutter button.

== Settings ==
Device settings, also referred to as "Preferences", allow improving the convenience of your usage. Settings to consider adjusting, where available, are:

* Enabling the LED indicator during stand-by mode to facilitate finding and picking up the smartphone in dark environments. The LED may blink or stay on continuously. Its power consumption is marginal only, making no noticeable difference in battery endurance.
* Enabling alternative methods for waking up from stand-by, such as double-tap, which takes less effort than pushing the power or home button. Some smartphones may allow waking up through the proximity sensor, which is referred to as "air wake-up" by Samsung, but works while the device is not moving to prevent accidental detection. Smartphones may allow automatically waking up when picked up.
* Specifying applications to be launched with touch gestures detected while the device is in stand-by mode.
* Deactivating transition animations, as they cause inconvenient delays of inertia. These include transitions between menu pages, unfolding the quick control menu, opening applications, and opening the list of recent tasks. On the popular Android operating system and its derivatives, this option is found in the "Developer options", which first have to be revealed by tapping the "build number" entry in the device information menu seven times.
* Deactivating remote volume control by Bluetooth clients such as vehicle audio systems, to be able to retain manual volume control on the smartphone. On Android OS and derivatives, this is also found in the developer options.
* Smartphones may implement gestures recognizable during stand-by mode, also known as "off-screen gestures" or "screen-off gestures". Such may include double-tapping to wake up the device from stand-by mode without having to push any button, and drawing shapes to quickly launch applications such as the camera.<ref>[https://thelitereview.com/phones/ulefone-s7-pro-review-2gb-ram-16gb-rom/ uleFone S7 Pro Review (2GB RAM & 16GB ROM) (2018-06-10)]</ref><ref>[https://www.androidauthority.com/oxygenos-features-879547/ ''OxygenOS: 6 features you need to know about'']</ref>

== Accessories ==
Various accessories could increase convenience and practicality, such as a kickstand case for watching video and typing (depending on angle), a cover, possibly one with preview display for quick assessment of notifications and messages and missed calls, a USB On-the-go adapter for plugging in USB flash drives, and a wireless charging station if supported.

Support for accessories varies among mobile phone models.

== Picking a smartphone ==
If you have difficulty deciding which smartphone to pick, ask yourself which desired functionality is missing from your current smartphone, and what your primary tasks are. If your budget is too low, weigh the importance of features against each other, and get the phone with the more important features.

== Decisions after purchase ==
You may wish to unlock the smartphone's bootloader in order to be able to "root" the device for [[:w:Rooting (Android)#Advantages|administrative tasks such as creating full backups and advanced modifications]]. However, this may void the warranty. It is recommended to make that decision soon after purchase, as switching the bootloader state may require a full erasure of user data, which becomes increasingly difficult to back up with accumulation over time.

== Apps ==
It may be necessary to install multiple apps with a similar purpose, such as file managers, as they functionally may complete each other.

== Screenshots ==
Depending on implementation screenshots may be captured with a short press or an approximately second-long hold of a button combination. The former is preferable, as accidental shots are better than missed shots. Accidental ones can simply be deleted if desirable, whereas missed ones can not simply be recreated.

== Further tips ==
* Do not overrely on a smartphone's water protection. See it as a protection against accidents, but put it into an additional water-protecting case when entering water.<ref>{{Cite web |url=https://www.droidviews.com/is-the-sony-xperia-z2-waterproof/ |first=Adit |last=Sharda |date=2014-05-15 |access-date=2022-01-06 |title=Is the Sony Xperia Z2 Really Waterproof? |language=en-US |website=DroidViews }}</ref>
* Mobile browsers usually allow viewing pages' source code by manually typing <code>view-source:</code> before the URL.<ref>{{cite web |title=How to View Page Source on Mobile browsers? |url=https://techglimpse.com/view-page-source-on-mobile-browsers/ |website=Techglimpse |access-date=6 January 2022 |date=2017-10-11}}</ref>
* Before applying a screen protector, clean it thoroughly using a micro fibre cloth to prevent ugly marks from grains of dust.
* Depending on vendor and configuration, it may be possible to launch the camera quickly to prevent missing moments by pressing the home button or stand-by button twice. Some smartphones feature a dedicated camera shutter button, which may allow for quick launching through double-pressing or holding.<ref>{{cite web |title=Sony Xperia Z2 camera shootout: feature focus |url=https://www.androidauthority.com/sony-xperia-z2-camera-shootout-374656/ |website=Android Authority |access-date=6 January 2022 |date=30 April 2014}}</ref>
* You may choose to photograph and film at lower resolutions to extend the remaining number of photos and recording duration in the remaining data storage space.<ref>{{cite web |title=How Do I Maximize Recording Time On My Recorder? |url=https://www.avertx.com/how-do-i-maximize-recording-time-on-my-recorder/ |website=www.avertx.com |access-date=6 January 2022}} (applies to any camera)</ref>

== See Also ==
* [[Should batteries in portable electronics be replaceable?]]
* [[Wikipedia: Smartphone]]
* [[Wikipedia: List of open-source mobile phones]]

== References ==
{{Reflist}}

[[Category:Smartphones| ]]

Linux/Troubleshooting

2025-01-04T20:42:27Z

Elominius: /* Unable to boot */ external editing

The goal of this page is to list solutions to common issues. You are encouraged to add yours here.



== Unable to boot ==
A possible cause of boot failure are entries to non-existent devices or image files in <code>/etc/fstab</code>.

Type in <code>/bin/nano /etc/fstab</code> in the emergency shell to edit the <code>fstab</code> file, then either comment (add <code>#</code> at the beginning of the line) or remove lines that contain entries to non-existent devices or image files.

You can also edit the <code>fstab</code> file from an external Linux installation (on a flash drive, an SSD, or a boot disc).

== 90-second delay while booting ==
Such a delay may be caused by an incorrect entry in the <code>/etc/fstab</code> file. This can especially be caused with [[:w:Live USB|live USB operating systems]] used across computers.

Enter <code>blkid</code> into the command line terminal, then compare the output with the contents of aforementioned file. One of the entries, such as the entry for the swap file (which contains the word "swap"), may need to be removed or commented out by adding <code>#</code> at the beginning of the line.<ref>[https://forums.linuxmint.com/viewtopic.php?t=327483 ''Boot proces 90 seconds delay'' – LinuxMint Forums, August 2020]</ref>

== sysrq ==
Some [[wikipedia:Magic_SysRq_key|system requests]] might be disabled out of the box due to the [[wikipedia:bitmask|bitmask]] value in the file <code>/proc/sys/kernel/sysrq</code> being set to a different value than 1, such as 176. A possible reason is to protect inexperienced users.

To override this, use the command <code>echo 1 | sudo tee /proc/sys/kernel/sysrq</code>, where the number "1" is piped into "tee", and <code>sudo tee</code> writes the "1" into said system file using root privileges.

Directly <code>sudo echo 1 >/proc/sys/kernel/sysrq</code> or <code>sudo printf 1 >/proc/sys/kernel/sysrq</code> does not work if the terminal is not running as root, since text redirection into the file is performed by the terminal, whereas with <code>sudo tee</code>, it is performed by the <code>tee</code> tool using root privileges.

== Mixing desktop environments ==
Your preferred desktop environment might lack an essential feature such as the popular "Cinnamon" desktop environment lacking a multiple-row task bar.<ref>[https://forums.linuxmint.com/viewtopic.php?t=162727 ''Enable multiple rows in the daskbar'' – LinuxMint Forums, March 2014]</ref><ref>[https://forums.linuxmint.com/viewtopic.php?t=239297 Resize panel/taskbar for two rows? - Linux Mint Forums, February 2017]</ref>

It is possible to mix parts of desktop environments, such as using the <code>mate-panel</code> while other parts like the task switcher of ''Cinnamon'' are still operational. The ''Cinnamon'' task bar can be set to hide automatically in the configuration, whereas the multiple-row ''Mate'' taskbar can be made to always display on screen.

In order to start multiple panels automatically, add one of them to the list of programs to start automatically. Typically, it is found in the system control panel, though the location may vary across desktop environments.

== Halting ==
If your computer halts ("freezes") frequently, the most common causes are tasks consuming high CPU or disk I/O usage. The former can be monitored with the command-line utilities including <code>top</code> (usually bundled), <code>htop</code> (more sophisticated but may require installation first), and the graphical <code>gnome-system-monitor</code>. Disk I/O usage can be monitored with the interactive command-line utility <code>iotop</code>, where the <code>-o</code> parameter hides inactive processes for a better overview.

A high I/O usage may not only caused by reads and writes themselves, but can also be caused by latency. This is not of concern on flash storages, but it is on hard disk drives, since the magnetic head needs an additional fraction of a second to move to the physical location of requested data.

== Halted window manager ==
If the cursor has become unresponsive, and you wish to get out of it without losing your session, try the following:

* Go to a different ''[[:w:tty (Unix)|tty]]'', for example, press Ctrl+Alt+F2 to go to tty2.
* Type the <code>w</code> command to list active sessions. Notice the content of the <code>FROM</code> column, possibly <code>:0</code>.
* Type <code>[process name of desktop environment] --replace --display=:0 2>/dev/null &</code>. Process names for desktop environments include <code>cinnamon</code>, <code>mate-panel</code> or <code>wm-mate</code> ("window manager MATE"), <code>xfwm4</code> (for xfce), and <code>kwin_x11</code> or <code>plasmashell</code> for KDE.<ref>[https://www.reddit.com/r/kde/comments/a5d2ly/how_do_you_properly_restart_kwin_and_plasmashell/ ''How do you properly restart KWin and Plasmashell from the virtual console?'' – /r/KDE, December 2018]</ref>

The <code>:0</code> part is from the "FROM" column in the prior command, the <code>2>/dev/null</code> part prevents diagnostic error messages from flooding the command line output, and the <code>&</code> character puts the process in background.<ref>[https://unix.stackexchange.com/questions/31818/what-to-do-when-a-linux-desktop-freezes ''What to do when a Linux desktop freezes?'' - Unix & Linux Stack Exchange]</ref>

== Password fail delay ==
Linux installations may be shipped with a long password fail delay, meaning an artificial delay until the user is allowed to retry after entering a wrong password.

While it is intended to prevent password cracking through brute-forcing, a delay that is too long, such as three seconds, could become annoying from repetition and interfere with workflow.

The fail delay is not recommended to be deactivated completely, but can be shortened to something more sufferable like 0.3 seconds. The disadvantage that brute forcing would become faster could be compensated multiple times over by appending one or two characters to the password. However, even at a fail delay of 0.01 seconds, brute forcing would be severely handicapped. As such, a preset fail delay of three seconds appears to be [[:w:Security theater|security theater]].

;Setting
There are multiple system files where the password fail delay may be configurable. Make sure to create a backup copy of of any system file before modifying.

The exact used system file may vary depending on operating system distribution and version, but the configuration is typically located at <code>/etc/pam.d/common-auth</code> or <code>/etc/pam.d/login</code>, and the delay value is specified in microseconds.

For example, to set the password delay to the tenth of a second, set the pre-existing value in the "login" file, which is presumably 3000000 out of the box, to "100000", as shown here:
<pre>
#
# The PAM configuration file for the Shadow `login' service
#

# Enforce a minimal delay in case of failure (in microseconds).
# (Replaces the `FAIL_DELAY' setting from login.defs)
# Note that other modules may require another minimal delay. (for example,
# to disable any delay, you should add the nodelay option to pam_unix)
auth optional pam_faildelay.so delay=100000
</pre>

Alternatively, try adding the new line <code>auth optional pam_faildelay.so delay=100000</code> to the "common-auth" file at this position:
<pre>
# here are the per-package modules (the "Primary" block)
auth optional pam_faildelay.so delay=100000
auth [success=1 default=ignore] pam_unix.so nullok_secure nodelay
</pre>

<ref>[https://unix.stackexchange.com/questions/2126/why-is-there-a-big-delay-after-entering-a-wrong-password security - Why is there a big delay after entering a wrong password? - Unix & Linux Stack Exchange]</ref>

== Black screen ==
On some proprietary graphics drivers, a black screen may occur under when switching away from the graphical user interface (usually tty7) to any other tty (Ctrl+Alt+F[number]) while screen resolution is not set to maximum does result in a black screen with only backlight visible, meaning the screen is still on but nothing is visible. This may also happen after waking up from suspending after a countdown, e.g. <code>sleep 1h; systemctl suspend</code>.<ref>https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-440/+bug/1870758</ref>

Adjust the screen resolution blindly by entering <code>xrandr -s 1920x1080</code> or another valid resolution in a terminal running inside X server. Note that for it to work, it has to be a different resolution than the current one. Consider adding <code>screenrescue() { xrandr -s 640x480; xrandr -s 1920x1080; redshift -O 2400; }</code> to <code>~/.bashrc</code>, to make a more memorable command named "screenrescue" do it.

== References ==
<references />

[[Category:Linux]]

Linux/Troubleshooting

2025-01-04T20:34:22Z

Elominius: "/etc/fstab" might prevent booting.

The goal of this page is to list solutions to common issues. You are encouraged to add yours here.



== Unable to boot ==
A possible cause of boot failure are entries to non-existent devices or image files in <code>/etc/fstab</code>.

Type in <code>/bin/nano /etc/fstab</code> in the emergency shell to edit the <code>fstab</code> file, then either comment (add <code>#</code> at the beginning of the line) or remove lines that contain entries to non-existent devices or image files.

== 90-second delay while booting ==
Such a delay may be caused by an incorrect entry in the <code>/etc/fstab</code> file. This can especially be caused with [[:w:Live USB|live USB operating systems]] used across computers.

Enter <code>blkid</code> into the command line terminal, then compare the output with the contents of aforementioned file. One of the entries, such as the entry for the swap file (which contains the word "swap"), may need to be removed or commented out by adding <code>#</code> at the beginning of the line.<ref>[https://forums.linuxmint.com/viewtopic.php?t=327483 ''Boot proces 90 seconds delay'' – LinuxMint Forums, August 2020]</ref>

== sysrq ==
Some [[wikipedia:Magic_SysRq_key|system requests]] might be disabled out of the box due to the [[wikipedia:bitmask|bitmask]] value in the file <code>/proc/sys/kernel/sysrq</code> being set to a different value than 1, such as 176. A possible reason is to protect inexperienced users.

To override this, use the command <code>echo 1 | sudo tee /proc/sys/kernel/sysrq</code>, where the number "1" is piped into "tee", and <code>sudo tee</code> writes the "1" into said system file using root privileges.

Directly <code>sudo echo 1 >/proc/sys/kernel/sysrq</code> or <code>sudo printf 1 >/proc/sys/kernel/sysrq</code> does not work if the terminal is not running as root, since text redirection into the file is performed by the terminal, whereas with <code>sudo tee</code>, it is performed by the <code>tee</code> tool using root privileges.

== Mixing desktop environments ==
Your preferred desktop environment might lack an essential feature such as the popular "Cinnamon" desktop environment lacking a multiple-row task bar.<ref>[https://forums.linuxmint.com/viewtopic.php?t=162727 ''Enable multiple rows in the daskbar'' – LinuxMint Forums, March 2014]</ref><ref>[https://forums.linuxmint.com/viewtopic.php?t=239297 Resize panel/taskbar for two rows? - Linux Mint Forums, February 2017]</ref>

It is possible to mix parts of desktop environments, such as using the <code>mate-panel</code> while other parts like the task switcher of ''Cinnamon'' are still operational. The ''Cinnamon'' task bar can be set to hide automatically in the configuration, whereas the multiple-row ''Mate'' taskbar can be made to always display on screen.

In order to start multiple panels automatically, add one of them to the list of programs to start automatically. Typically, it is found in the system control panel, though the location may vary across desktop environments.

== Halting ==
If your computer halts ("freezes") frequently, the most common causes are tasks consuming high CPU or disk I/O usage. The former can be monitored with the command-line utilities including <code>top</code> (usually bundled), <code>htop</code> (more sophisticated but may require installation first), and the graphical <code>gnome-system-monitor</code>. Disk I/O usage can be monitored with the interactive command-line utility <code>iotop</code>, where the <code>-o</code> parameter hides inactive processes for a better overview.

A high I/O usage may not only caused by reads and writes themselves, but can also be caused by latency. This is not of concern on flash storages, but it is on hard disk drives, since the magnetic head needs an additional fraction of a second to move to the physical location of requested data.

== Halted window manager ==
If the cursor has become unresponsive, and you wish to get out of it without losing your session, try the following:

* Go to a different ''[[:w:tty (Unix)|tty]]'', for example, press Ctrl+Alt+F2 to go to tty2.
* Type the <code>w</code> command to list active sessions. Notice the content of the <code>FROM</code> column, possibly <code>:0</code>.
* Type <code>[process name of desktop environment] --replace --display=:0 2>/dev/null &</code>. Process names for desktop environments include <code>cinnamon</code>, <code>mate-panel</code> or <code>wm-mate</code> ("window manager MATE"), <code>xfwm4</code> (for xfce), and <code>kwin_x11</code> or <code>plasmashell</code> for KDE.<ref>[https://www.reddit.com/r/kde/comments/a5d2ly/how_do_you_properly_restart_kwin_and_plasmashell/ ''How do you properly restart KWin and Plasmashell from the virtual console?'' – /r/KDE, December 2018]</ref>

The <code>:0</code> part is from the "FROM" column in the prior command, the <code>2>/dev/null</code> part prevents diagnostic error messages from flooding the command line output, and the <code>&</code> character puts the process in background.<ref>[https://unix.stackexchange.com/questions/31818/what-to-do-when-a-linux-desktop-freezes ''What to do when a Linux desktop freezes?'' - Unix & Linux Stack Exchange]</ref>

== Password fail delay ==
Linux installations may be shipped with a long password fail delay, meaning an artificial delay until the user is allowed to retry after entering a wrong password.

While it is intended to prevent password cracking through brute-forcing, a delay that is too long, such as three seconds, could become annoying from repetition and interfere with workflow.

The fail delay is not recommended to be deactivated completely, but can be shortened to something more sufferable like 0.3 seconds. The disadvantage that brute forcing would become faster could be compensated multiple times over by appending one or two characters to the password. However, even at a fail delay of 0.01 seconds, brute forcing would be severely handicapped. As such, a preset fail delay of three seconds appears to be [[:w:Security theater|security theater]].

;Setting
There are multiple system files where the password fail delay may be configurable. Make sure to create a backup copy of of any system file before modifying.

The exact used system file may vary depending on operating system distribution and version, but the configuration is typically located at <code>/etc/pam.d/common-auth</code> or <code>/etc/pam.d/login</code>, and the delay value is specified in microseconds.

For example, to set the password delay to the tenth of a second, set the pre-existing value in the "login" file, which is presumably 3000000 out of the box, to "100000", as shown here:
<pre>
#
# The PAM configuration file for the Shadow `login' service
#

# Enforce a minimal delay in case of failure (in microseconds).
# (Replaces the `FAIL_DELAY' setting from login.defs)
# Note that other modules may require another minimal delay. (for example,
# to disable any delay, you should add the nodelay option to pam_unix)
auth optional pam_faildelay.so delay=100000
</pre>

Alternatively, try adding the new line <code>auth optional pam_faildelay.so delay=100000</code> to the "common-auth" file at this position:
<pre>
# here are the per-package modules (the "Primary" block)
auth optional pam_faildelay.so delay=100000
auth [success=1 default=ignore] pam_unix.so nullok_secure nodelay
</pre>

<ref>[https://unix.stackexchange.com/questions/2126/why-is-there-a-big-delay-after-entering-a-wrong-password security - Why is there a big delay after entering a wrong password? - Unix & Linux Stack Exchange]</ref>

== Black screen ==
On some proprietary graphics drivers, a black screen may occur under when switching away from the graphical user interface (usually tty7) to any other tty (Ctrl+Alt+F[number]) while screen resolution is not set to maximum does result in a black screen with only backlight visible, meaning the screen is still on but nothing is visible. This may also happen after waking up from suspending after a countdown, e.g. <code>sleep 1h; systemctl suspend</code>.<ref>https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-440/+bug/1870758</ref>

Adjust the screen resolution blindly by entering <code>xrandr -s 1920x1080</code> or another valid resolution in a terminal running inside X server. Note that for it to work, it has to be a different resolution than the current one. Consider adding <code>screenrescue() { xrandr -s 640x480; xrandr -s 1920x1080; redshift -O 2400; }</code> to <code>~/.bashrc</code>, to make a more memorable command named "screenrescue" do it.

== References ==
<references />

[[Category:Linux]]

File management

2024-12-13T10:22:40Z

Elominius: Trash bin (recycle bin) introduction

This lesson teaches the management of digital files and directories.

While there exists no ideal method of file management, this resource documents possibly helpful practices and inspiring ideas.

This lesson was created under the impression that the average computer and mobile phone user still struggles to keep track of files in the long term despite of all the tools at their disposal.

== Descriptive file names ==
Every file manager has a "rename" feature, letting you change the names of files and folders.

Don't just let this essential feature "rust in the junk yard". Utilize it to make files easily searchable. For example, add a short description after a picture or video file name. If you recorded using a [[smartphone]], you can rename the file on the device itself right afterwards. For example,

* <code>VID_{{#time:Ymd_His}} trampoline jumping.mp4</code>

It is recommended to add the short description after the file name rather than before, to prevent interference with alphanumerical sorting. Even though file managers can still sort items by date and time, files' time stamp attribute might not always be retained throughout transfers, for example when transferring files through [[:w:Media Transfer Protocol|MTP]], between a smartphone's internal memory and memory card, or uploading files to a cloud storage service. A time stamp at the beginning of the file name allows alphanumerical sorting to act as chronological sorting even if the date and time attribute is lost.

Not every single file needs to be given a name manually, since that would take much effort. If your file manager lacks range selection and bulk renaming, meaning the selection and renaming of many items at once, adding a descriptive file name to the first file of similar files will still facilitate finding those files, since the title of the first file in the group covers the remaining files. For example, all four of these imaginary files depict strawberries, but only the first was manually named:

* <code>IMG_20230914_090112 strawberries.jpg</code>
* <code>IMG_20230914_090116.jpg</code>
* <code>IMG_20230914_090119.jpg</code>
* <code>IMG_20230914_090124.jpg</code>

== Directories ==
Make use of directories. Categorize files into logically structured directories to facilitate finding them later.

Some inexperienced computer users might have a habit of just throwing files into the root directory of their device such as a flash drive. With an increasing count of accumulating files, they would become increasingly difficult to find, especially multimedia content which can not simply be searched for text strings using tools like `grep` but only through metadata.

Create a mind map by asking yourself where you would look for these files whenever necessary.

Below are two example directory structures for video projects, where the project files of the first project are directly in the project folder, while those of the second are in a subfolder; a matter of preference.

<blockquote>
* Video projects
** Example project 1
*** Assets
** Example project 2
*** Project files
*** Assets
</blockquote>

Structure files in the order your mind recalls them. For example, the type and source of files is more memorable than the date. For more examples of directory structures, see the [[/directory structures]] sub page. Feel free to add examples to it.

For directories where a sufficiently descriptive directory name would be considered too long, consider creating a descriptive text file inside named <code>description.txt</code>, <code>info.txt</code>, or similar. Comments about specific files may be noted in a text file with <code>.meta</code>, <code>.meta.txt</code>, or a similar suffix appended to its name.

If files within the same category have wildly varying sizes, such as command line outputs, you might want to move files beyond a size threshold such as 1 MB or 5 MB into a separate folder whose name can be the same with an added "-large" suffix. This allows minimizing the size of compressed snapshots of the directory.

Avoid putting sub-directories in directories populated with many files, since they can become a nuisance when wanting to view the lastest file whereas the file manager lists sub-directories at the top, or otherwise be difficult to find between the files when needed. Due to the slow loading times of the [[:w:Media Transfer Protocol|Media Transfer Protocol (MTP)]] which is commonly used to connect mobile devices to desktop/laptop computers, and due to the possibility of file managers trying to determine file types by reading internal file information,<ref>''[https://github.com/linuxmint/nemo/issues/1907 nemo significantly slower opening folder content #1907 - GitHub]'' ([[:w:Nemo (file manager)|Nemo file manager]] reads the [[:w:Header_(computing)|header]] of each file with unknown [[:w:Filename extension|extension]]</ref> it could make navigating more difficult. Therefore, it should be considered to create a separate folder that only contains subfolders, such as <code>Downloads_2</code>.

You don't need to spend much effort coming up with good names for files and folders, as you can change them later at any time if you happen to come up with a better idea.

== Revision history ==
When spending much time on a project, write-up, email draft, etc., consider saving a revision history, by selecting "Save as" (also accessible through <kbd>Ctrl</kbd>+<kbd>↑Shift</kbd>+<kbd>S</kbd> keyboard shortcut on some software) and changing the file name through numbering or timestamping, or alternatively creating renamed copies externally using a file manager or command terminal. For possible file naming schemes and variations of numbering and time-stamping, see [[File naming]].

This enables you to revert to an earlier revsion in case of error, prevents total loss in case of failed writes caused by a power outage or software crashing, and later facilitates comprehending the work progression, while only consuming marginial space compared to common data storage, and being efficiently [[Backup#Compressed_archives|compressible]] due to redundancy.

A new revision does not necessarily have to be created upon every saving, but whenever the changes since the last revision are major enough at your discretion. Optionally, a short comment summarizing the changes can be added to the file name.

For programming code, suffixes like <code>-stable</code> and <code>-unstable</code> may be added after the time stamp, for example: <code>UserScriptName-revision-{{#time:YmdHis}}-stable.js</code>. Separating the number or time stamp allows convenient double-click selection in the file saving dialogue.

Revisions can optionally routinely be moved into a separate subfolder.

== Dumping ground ==
For files and folders you are unsure where to put, consider creating a directory on your device named <code>dump</code>, <code>sandbox</code>, or similar.

You may wish to categorize those into text, compressed archives, drawings, or by whichever task, or dated [[file naming|folder names]] such as <code>{{#time:Y-m-d}}</code>.

When managing files and directories in a command line terminal, an item can temporarily be given a simple and short name such as <code>1</code> to facilitate typing in the commands, that will be changed shortly after. Examples are:

; Output from multiple commands into one log file:
<syntaxhighlight lang=sh>
$ ls [path] -alR >>1 # Write names and attributes of that folder's content into the file "1".
$ find [path] >>1 # Write a list of bare file paths into the file "1".
$ mv -n 1 [desired file name] # Rename "1" to desired file name. "-n" prevents accidental overwriting.
</syntaxhighlight>

The above command could also be done in a single line with <code>(ls ''[path]'' -alR; find ''[path]'') >>''[desired file name]''</code>, but the former might be preferred to save time if the target directory contains a high number of files, as the former command starts immediately after pressing <kbd>⏎ Return</kbd> (also known as "Enter"), or when one wishes to enter commands before thinking of an output file name, while the latter requires typing in the whole command first.

; Moving files from multiple sources into a directory:
<syntaxhighlight lang=sh>
$ mkdir 1 # Create a folder named "1".
$ mv -n *.mp4 *.mkv 1 # Move mp4 and mkv files into it.
$ mv -n 1 Videos-$(date +%Y-%m-%d) # Renaming folder to intended name.
</syntaxhighlight>

Should it be necessary to print out the absolute (full) file path of a file for, for example, quick copying to the clipboard, use the command <code>readlink -f ''[target file]''</code>, and for information about the current mount point, use the <code>findmnt -T .</code> command.

== Finished tasks ==
Files whose task has finished, such as a posted message that was initially drafted offline into a text file, can be moved into a subfolder of the current directory named <code>done</code>, or alternatively in one big shared folder for this purpose.

== Storage types ==
Auxiliary data which is frequently accessed can reside on the operating system drive, on its own partition or a separate one. This especially applies to portable computers and operating systems installed on external media such as a USB flash drive.

=== Secondary and external storage ===
On desktop computers, large files can reside on a secondary large hard disk drive or solid state drive, the former of which costs less per space. For laptops, stationary hard drives located at home or portable external hard drives or solid state drives can be used. While working with a laptop as passenger in any moving vehicle such as a bus or tram, solid state memory is preferrable due to sturdiness, as hard disk drives do not like constant physical movement due to mechanical wear. If portable storage necessary, files can be stored on a constantly inserted memory card, which does not compromise ergonomy as they don't physically protrude, or at most a little. In addition, it can be occasionally removed when the data is needed on a different device. SD cards with 1 TB exist since at least 2017, though are expensive.<ref>[https://www.engadget.com/2016-09-20-sandisk-1tb-sd-card.html ''SanDisk outs the 'world's first' 1TB SD card''] (2016-09-20)</ref>

A home server adds the convenient benefit of access from all devices at home, and even through the internet if set up by the user ("private cloud"), but typically has longer latency times (access delays) than physically attached storage, and lacks mass storage access that may be necessary for some programs to work properly. In such a case, the files would have to be downloaded first and worked with locally, and uploaded after finishing.

=== Incubate work on flash storage ===
If, for example, your computer has a setup with solid state drive for operating system and hard disk as expansion storage, a project may be worked on on the flash storage, and changes can be applied to the hard drive at the end of a day.

If the hard drive is set to spin with no or a long timeout such as an hour or more, this may not be as necessary, but for short timeouts, frequent spin-ups would cause mechanical wear and tear in addition to annoying delays.

Using flash memory can be particularly useful on battery-powered laptops for power efficiency, though mechanical hard drives are increasingly being usurped by solid state memory in laptops, mainly due to physical robustness, as hard drives on laptops have long served for cost saving, whereas solid state memory is becoming increasingly affordable since the mid-2010s, though external USB hard drives may be used on the go.

=== Hard drives' purposes ===
After purchasing a hard drive, choose whether to dedicate it to either auxiliary storage or archival ([[wikipedia: cold data|cold storage]]).

Use as auxiliary storage such as for a workstation or server demands the drive be in constant operation, which wears it down over time, making it unsuitable for long-term archival.

Use as archival storage demands only sporadic (rare) operation to add or retrieve data, which induces far less mechanical wear.

Data retrieved from an archive drive should be copied over to auxiliary storage to avoid needless wear on the former whenever that data is recalled again in near future, as can be expected, and in order to further duplicate files that have proven to be useful, as archival is commonly done with an uncertainty of which files will be useful in future. A functional archive drive may later be repurposed for auxiliary storage after moving the archived data over to a new device.

Make a list of data you wish to retrieve the next time you access your archive drive. If you can afford it, mirror the archive to a readily available hard disk or network-attached storage. If your live hard disk or network attached storage runs out of disk space, delete the files that were not accessed for the longest time if they are already backed up somewhere else. To find them, file managers are equipped with the ability to sort files and directories by "last accessed" (or similar) in either direction.

=== Labels ===
Consider physically labeling your storage media to facilitate finding data. For example, a date or a short summary of the contents can be written on a label sticker that is applied onto the casing of an external data storage device.

For optical discs, a disc marker (also interchangeably known as "CD marker", "DVD marker", etc.) can be used to write notes directly on the disc. If more space is necessary, label stickers on its containing [[:w:Optical_disc_packaging#Jewel_case|jewel case]] or spindle can be used. Many jewel cases include a paper sheet for notes. Do not attach stickers directly onto the disc, as they could disintegrate during high rotation speeds, risking damage to the optical drive's internal components.

It is also recommended to change the device's file system label to a year or a short summary of the contents so it can quickly be seen in the file manager's device list (usually a side bar on the left) when plugged in.

On data storage devices whose life expectancy appears to be nearing its end, as indicated by [[:w:Self-Monitoring,_Analysis_and_Reporting_Technology|S.M.A.R.T. data]] on hard drives and performance loss on solid state memory. an "expired" or "EOL" (end of life) label can be added. Such devices should at most be used for temporary purposes such as testing.

== Data storage size ==
It is recommended to get a larger storage capacity than one intends to use. This leaves some spare room in the case that more data than expected is created, such as vacation video recordings.

On portable devices such as mobile phones and digital cameras, a larger storage capacity lowers the required frequency of file transfers into ones archive.

Another benefit of larger storage media is that more data can be readily accessible at a given time, and a needed file can be found faster since fewer devices have to be searched. However, a central file list and [[#Labels|labeling]], as described in other sections, can facilitate finding files as well.

If your storage device has, for example, 128GB, it realistically means 100 GB. Not only because of reserved space from file system overhead and possibly operating system files, but because of those 128 GB, the end is less useful, since one might want to put files there that are slightly larger than the remaining space, and one would like to have all those files in one location to avoid confusion and the annoyance of changing media.

In other words, if one would like to put 11 GB where only 10 GB are available, those 10 GB are as little useful as 0 GB.

== Partitioning ==
User data may be stored on the same or a separate partition as the operating system.

The benefit of a separate partition for user data is that possible file system corruption on the operating system partition would not spill over to user data, though modern file systems such as ''NTFS'' and ''ext4'' protect themselves from damage by [[:w:Journaling file system|journaling]], which allows the file system driver to recover quickly after an unexpected termination of write access caused by an operating system crash or unexpected removal.
Infrequently accessed files that may be necessary in near future can be stored there as well, or moved to a secondary drive if their size is significant.

Another benefit of a separate user data partition is the smaller backup size of the operating system partition and facilitated recovery in case of a malfunctioning operating system where other means of repair have failed or would be too difficult. Because operating systems are subject to corruption and can at worst become unbootable, it is good practice to [[backup|back up]] their partition regularly into a disk image. A smaller operating system partition can be imaged more quickly and the routine induces less wear on the backup media. Should the operating system malfunction, it can be imaged and then restored more quickly from the functional previous disk image, with less work to merge desirable changes since the last backup.

== Packing and archival ==
Perpetual streams of new files such as web downloads, photos and videos from digital cameras and cell phones, and screen captures can be packed by renaming their parent folder into a uniquely identifying name, such as with date stamp: <code>Camera-{{#time:Y-m-d}}</code>, after which they can be moved to an archive drive at the next backup appointment. The folder's name may also contain a location, device type, and/or short description. If packed more than once on the same day, time stamps or part numbers can be added to the names, or the files can be merged into the same directory.

If you no longer intend to change the contents of an archived folder, a file count and byte size can optionally be added to the name to provide a quick overview in file lists and to facilitate verifying whether all data has been transferred without having to navigate back to the source device and wait for loading to view the folder's byte size. An example name is <code>Camera-{{#time:Y_m_d}}-12147367878b-2093items</code>. It is recommended to use the exact byte number to prevent confusion between file size units that are powers of two (KiB, MiB, GiB) and powers of ten (KB, MB, GB).

Alternatively to renaming on the source device, the directory with uniquely identifying name can be created on the archive drive first, and files can be moved there out of the source directory.

When to pack files is end users' decision, though it is recommended to do so before exhausting free space on the source device. Renaming is not as necessary if new folders are created automatically, like [[:w:Digital_camera#Directory_and_file_structure|some digital camera firmwares do per 999 or 1000 pictures]]. All filled folders can be considered eligible for archival.

Larger storage space provides the benefit of more buffer until the next file transfer becomes necessary to clear space, thus it needs to be done less frequently.

Individual files needed for a specific purpose such as an impending project can be copied or moved into a separate directory.

== Write protection ==
Write protection may be desirable to defeat the fear of accidental modification of data when not desired by the user.

A simple way of achieving write protection in Linux-based operating systems is to mount or re-mount a device or partition as read-only with this command which requires superuser privileges: <code>mount -o remount,ro ''[device or mountpoint]''</code>.

If write-protection is not supported by the operating system, an SD card with write protection switch feature can be used. The switch relies on the SD card reader to obey it and deny writing access to the operating system. Some memory card readers, both built-in ones and USB adapters, might not obey the write protection switch.

Another way to achieve write protection is finalized write-once optical media or a read-only optical drive with insufficient laser beam power to write data, as described in {{section link||Sensitive environment}}.

== File listing ==
Searches within file lists inside a text file are significantly faster than searches through a file system.

See [[Data_recovery#Create_a_file_list|this guide]] on how to create file lists.

=== File index ===
As explained in {{section link|File_puzzling#Orphaned_directories}}, some file systems store directories that comprise file paths as "linked lists", meaning distributed over the entire space rather than one index of "nodes" at the beginning, which has both benefits and disadvantages, the latter of which is slower file searching.

A searchable file index stored in a text file named "index" created using the <code>find >index</code> command can facilitate finding files, as it contains a list of paths to all files at one place. The index can be updated by running the command again to overwrite the existing one. If the working directory is not the root of the file system, it should be changed to it or the paths need to be specified.

The "index" file can be searched with ease using <code>grep -i "searched file name" index</code>, which is typically much faster than directly searching the file system. <code>-i</code> may be left out for case-sensitive search. These commands have additional options, but these are outside the scope of this section.

== Time stamp preservation ==
Some methods of file transfer, such as copying within/onto mobile phone storage, the <code>cp</code> command without activated <code>-p</code> ("preserve") option, and a directory on Unix/Linux not owned by the current user, might discard date and time stamp file attribute(s), resetting it to the current time.

To preserve last-modified time stamps over FTP, downloading is preferred, as uploading while preserving it requires both client and server to support the [[:w:FTP#Additional_commands|MDTM (''Modify Fact: Modification Time'') command]], which it is not widely.

== High numbers of small files ==
With an increasing number of files, file searches slow down. High numbers of small files also restrict portability, as they demand more file operations for file transfers, slowing the process down.<ref>Related: [https://www.quora.com/Why-is-copying-1-000-1MB-files-so-much-slower-than-copying-1-1GB-file-given-that-the-same-amount-of-data-is-being-copied/answer/Franklin-Veaux Answer on Quora to "''Why is copying 1,000 1MB files so much slower than copying 1 1GB file, given that the same amount of data is being copied?''" by Franklin Veaux, on January 14th, 2020]</ref> Additionally, higher cluster sizes in combination waste more space to [[File Systems#Cluster_size|cluster overhead]] (unused reserved space).

If you happen to have a high number of currently unneeded small files, such as tens of thousands, consider packing them into one big archive file for improved portability.

Compression may be considered where efficient, such as in human-readable text files and code, and/or where more necessary, such as online file sharing. Compression ratios of 100 may be achievable by strong compression algorithms on text documents and code. However, it should be taken into consideration that damage magnifies enormously over compressed archives, as demonstrated in {{section link|Backup#Compressed archives}}. Therefore, it is recommended to store compressed archives on at least two devices.

Text inside compressed archives can be searched through directly without extracting using tools such as <code>zgrep</code>, <code>xzgrep</code>, <code>bzgrep</code>, and for 7-Zip, <code>7z e -so -bd ''[path]'' |grep ''[query]''</code>.

If frequent modification of small files is necessary, an alternative to packed archive files (such as <code>.tar</code>, <code>.zip</code>) are file system containers (''virtual disks'') with a small cluster size. File system containers are manually generated disk image files that can be mounted like usual drives.

== Avoid exhausting the operating system's partition ==
Exhausted space storage should be avoided, especially on an operating system partition, as it could lead to bogus behaviour by software not designed to handle such condition, or other unwanted behaviour. For example, a failed write while saving could blank the target file, causing the loss of work and reset of configuration. A web browser might [[Backup#Browsing_history|automatically delete early browsing history entries]] to make space for new. Even seemingly basic features such as command line parameter completion could malfunction.<ref>{{cite web |url=https://unix.stackexchange.com/questions/277387/tab-completion-errors-bash-cannot-create-temp-file-for-here-document-no-space |date=2016-04-18 |access-date=2022-02-10 |title=Tab completion errors: bash: cannot create temp file for here-document: No space left on device |website=Unix & Linux Stack Exchange }}</ref>

On an operating system partition, keeping a safety margin of free storage such as 5% at any time is recommended, and at least 1% on secondary expansion storage. On archival media, a controlled exhaustion of space is less critical, though the readability of the final written files should be verified.

Should you still find yourself with 100% exhausted space, first seek few megabytes of files to move out, perhaps temporarily, to improve system stability. This allows you to take time to calmly search for more and/or larger files to move out.

== File system repairs ==
Tools such as <code>CHKDSK</code> on Microsoft Windows and <code>fsck</code> on Linux promise repairing damaged file systems. Logical file system errors may be caused by unexpected power outages or unpluggings.

Be careful with file system repairs. It is recommended to back up any device (either to a full-disk image or by copying all files) prior to running one, in order to be able to revert with ease in case of unwanted collateral damage.

Detected file system errors may be caused by incompatibile file names across operating systems. For example, Linux allows characters in file and folder names in NTFS that Windows considers invalid, such as a colon (<code>:</code>) and a pipe character (<code>|</code>), as well as case-insensitive file names. Upon detection of invalid characters, the <code>CHKDSK</code> tool moves and renames such items, which leads to the loss of file names and paths. In particular, <code>ChkDsk</code> moves files and folders with invalid names into a directory located at the file system's root named <code>found.000</code>, and renames them to generic names like <code>file00000000.chk</code> and <code>dir_00000000.chk</code>, where the number is hexadecimal and incremented.

== Disk usage analysis ==
[[File:GNOME_Disk_Usage_Analyzer_3.32_screenshot.png|thumb|Disk usage analyzers facilitate finding directories with the largest content size. Some illustrate the directory structure graphically.]]
Disk usage analyzers calculate the size of directories on any selected path, allowing the user to easily discover directories which occupy the most space. Large folders not currently needed can be moved over to an archive drive, which clears the most space on the source device.

Popular tools for desktop operating systems include ''[[:w:Disk_Usage_Analyzer|Baobab]]'' for Linux (pre-installed on some popular distributions) and ''Xinorbis'' for Windows, both with sophisticated graphical user interfaces. Linux is also equipped with the command-line tool <code>du</code>, which allows outputting results directly into a text file.

For mobile (Android OS), [[:w:ES_File_Explorer|ES File Explorer]] is equipped with such functionality, though that application has been subject to controversy and has developed into adware.
<div style="float:none; clear:both;"></div>

== Deduplication ==
Several tools to automatically deduplicate files exist: rdfind, fdupes, jdupes, rmlint, dupeGuru, FSlint.

Storing one duplicate of files may be desirable in certain situations, such as [[Backup#Compressed_archives|compressed archives]] intended for long-term preservation, where even the slightest damage can render any data after the point of damage unreadable. For accessing the same file from different locations, [[:w:Hard_link|hard link]]s or [[:w:symbolic link|symbolic link]]s can be used.

You may want to deduplicate files across two different storage devices without accidentally deleting any files that do not exist on the storage device you want to keep the files on. A manual deduplication may be necessary if you wish to free up space from a device after copying files to a different device or after creating an archive file from those files without accidentally deleting any files that were not copied or archived from the source device, or after a file copying operation was interrupted and you need to clean up without deleting any files that were not copied.

The quickest way to accomplish this would be to compare the total count and size of those files in both the source and the target device by selecting the files and opening the "Properties" window. If the byte counts match, the source folder can be safely deleted. However, if the selection contains subdirectories too, differences in file systems mean that the total size might not match due to differences in how directories are stored. Some file managers also don't show the exact byte count.

If you want to deduplicate files between two different computers or between a computer and a smartphone, an additional problem is that file managers count files and folders differently. Some file managers only show the number of "items", meaning both files and folders, where as others may only show the number of "files" without counting the folders. If possible, use the same file manager on both devices, and one that shows the exact byte count.

A more sophisticated way to manually deduplicate files is creating a temporary script that contains a list of files to be deleted. In Linux, this can be accomplished first changing the working directory to the target folder (where the files are supposed to stay) using <code>cd</code> and then creating a temporary script by running:

<syntaxhighlight lang=sh>
# list files to be deduplicated
find -type f |sed -r "s/(.*)/rm -v '\1'/g" >>temporary.sh
# list empty directories to be removed
find -type d |sed -r "s/(.*)/rmdir -v '\1'/g" >>temporary.sh
# script deletes itself after work done
echo "rm temporary.sh" >>temporary.sh
</syntaxhighlight>

No files have been deleted up to this point. Only the disposable script to do the work has been created. This script contains a list of files to be deleted from the source device, but the list is generated from the target device to avoid containing the name of any files that have not been copied. It is recommended to use a temporary script rather than throwing around commands involving an <code>|xargs</code> pipe, since running <code>|xargs</code> commands in combination with deletion is dangerous since you don't see a list of files before starting the deduplication process.

The <code>-v</code> flags are optional and serve to later show the names of the files deleted in the deduplication process in the terminal.

Now move the <code>temporary.sh</code> file to the original directory on the source device from which you want to delete the duplicate files. Then change your terminal's working directory to that directory using <code>cd ''[path]''</code>. Before running the script, open it in a text editor and glance over it to verify that it doesn't contain any files you don't intend to delete. Then run it using <code>sh temporary.sh</code>.

Do not run the script before changing your working directory to the same directory the script resides in, because the script contains relative paths rather than full (absolute) paths, so the path of your terminal's working directory will be presumed to be infront of the relative paths in the script.

== Temporary folder ==
Programs may use temporary folders such as <code>/tmp/</code> and <code>~/.cache/</code> on Linux and <code>%temp%</code> on Windows to store data such as preview thumbnails and data from the web to reduce loading times.

Should the way your operating system or file manager handles its [[:w:Temporary folder|temporary]] or [[:w:Trash (computing)|trash folder]] not suit your needs (e.g. if retention span is unchangeable or limited), you may wish to manually operate such in your user home folder (e.g. <code>~/tmp/</code>, <code>~/trash/</code>). Some software may allow changing the path for the temporary folder. Furthermore, such folders can be used for frequent short-term backups of projects, which can be deleted when free space becomes necessary.

In comparison to a traditional "Trash" folder implemented by the operating system or file manager, files can be opened directly as usual, whereas a file manager may disallow directly opening files sitting in a "Trash" folder and show file properties instead, as Windows Explorer does, whereas, for example, the ''Nemo'' file manager for Linux allows directly opening files located in the trash directory.

== Trash bin (recycle bin) ==
When deleting files, you might want to consider using the [[:w:Trash (computing)|trash bin]] (or "recycle bin" and other names) feature, which stores files in a temporary location until they are automatically deleted. This allows reversing unwanted deletions, for example if you selected a file or folder you didn't mean to delete. This lifts the emotional burden and carefulness during deletion, since you know you can undo the deletion for the time being. Without it, deletions are "walks on eggshells", metaphorically speaking.

A recycle bin also makes file moving (with copying and deletion from source) safer, because during the deletion step, you have to be less careful to select the exact same files and folders that you moved. If your trash bin does automatic deletion after a certain time, you don't have to bother with cleaning it up later anymore, while you can still bring back the file if necessary in the near future. The files still take up space on your device while residing in the trash bin, but it doesn't matter if you have enough space free.

== Mobile ==
=== Memory cards ===
[[File:Huawei_U8950D_no_cover.JPG|thumb|right|Smartphone with inserted memory card (located below camera lens)]]
Some smartphones and tablet computers allow the expansion of storage capacity using memory cards, typically MicroSD, which significantly facilitates file management and is user-friendly.

Memory cards can be re-used immediately between devices without need for file transferring, and data stored on the memory card is not at the risk of mobile devices' technical defect, as it can be ejected, after which data can be retrieved externally. Mass storage access from an external computer also may allow [[data recovery|recovering some files]] imminently after a deletion accident caused by bogus software<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/ Users report Android devices' entire internal user storage being deleted instantly, caused by poor software design – "''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?''" – Reddit.com/r/Android (2013-01-11)]</ref> and/or human error.

For huge file transfers, ejecting the memory card and directly transferring to the PC through mass storage may save time compared to MTP (media transfer protocol) through the phone or tablet, as the latter does not handle high counts of files within a directory well. Additionally, memory cards can immediately be reused in a different device without lengthy file transfers. USB-OTG (''On The Go'') may be used as well, connected through an adapter directly to the mobile device, though it might not [[#Time_stamp_preservation|preserve a date and time attribute]]. Tablet computers with desktop operating system are widely equipped with at least one default-sized USB-A port.

Additionally, using a memory card takes stress off the device's non-replaceable internal memory, preserving its limited rewrite cycles, which is especially beneficial for repeated heavy tasks such as high-resolution filming and mobile FTP server hosting.
<div style="float:none; clear:both;"></div>

=== Between computer and mobile ===

==== Media Transfer Protocol ====
File management on mobile phones and tablet computers with mobile operating systems is more restricted than on desktop/laptop computers and mass storage devices such as USB sticks and memory cards, as the media transfer protocol (MTP), which is used to access files on a mobile device from a computer, lists files slowly, which is problematic for loading directories with high counts of files.

As such, it is recommended to manage such directories on the device itself. If transfer between a desktop/laptop computer is desired, handle those files in a little to unpopulated directory.

MTP file listing can be sped up by not loading preview thumbnails. Depending on file manager used, this may be done by deactivating preview thumbnails in settings or choosing the "detail" view mode, where files appear in a list instead of a grid which foregoes the loading of preview thumbnails. However, files can be dropped into a directory without opening the target directory, by pasting them through the right-click context menu.

A benefit of MTP is it not being prone to file system corruption as a result of unexpected removal, meaning without being "safely unmounted" through the client operating system (i.e. desktop / laptop computer), as it operates through an abstraction layer and the file system is controlled by a driver on a battery-powered host device (i.e. smartphone / tablet).

Only a selection of files, no directories, should be moved away from the device, because users have reported files on MTP not being listed properly.<ref>[https://android.stackexchange.com/questions/136839/files-show-up-on-nexus-5-but-not-in-windows-7 ''Files show up on Nexus 5 but not in Windows 7'' – Android Stack Exchange – February 11th, 2016]</ref><ref>[https://android.stackexchange.com/questions/46315/not-all-files-are-visible-over-mtp ''Not all files are visible over MTP'' – Android Stack Exchange – May 29th, 2013] (209,502 views as of September 21st, 2022)</ref><ref>[https://stackoverflow.com/questions/13737261/nexus-4-not-showing-files-via-mtp ''Nexus 4 not showing files via MTP'' – StackOverflow – December 6th, 2012] (67,504 views as of September 21st, 2022)</ref> If a directory is moved away from the device, the computer might delete it from the mobile device without all content having been transferred away. Instead, it should be copied, and the byte size be compared on both the computer and the smartphone itself, where a match indicates a successful transfer, meaning the directory can now be deleted from the mobile device. The only exception where moving folders out is safe is when the number of files within is overseeable, i.e. less than ten, where all files are clearly listed in the computer's file manager.

Windows Explorer additionally displays files while listing is in progress, which can be of use when moving files out, since the loading of the file list can be interrupted to allow moving out the displayed files chunk-wise, reducing the number of remaining ones each time.

Transferring files onto the device through MTP [[#Time_stamp_preservation|may dismiss their date and time attribute]].

If a file has newly been created on the smartphone while it was connected to the computer through MTP, the computer's file manager could potentially misreport the file size as too small due to having loaded the directory listing in a moment where the file was not complete. Moving the file away from the phone could cause it to be truncated (incomplete) on the target path while being deleted from the source, since the file manager might wrongfully assume that the file has been fully transferred while it hasn't.

==== File Transfer Protocol ====
An alternative to MTP is FTP (file transfer protocol) through ethernet.

On the desktop computer, a dedicated and sophisticated FTP client such as FileZilla (open-source) may be used to handle high numbers of files, though FTP is widely supported by file managers and web browsers.

FileZilla does not support moving files out of an FTP server, meaning downloading and deleting automatically, whereas moving within a server is supported through the standard rename command. If the intention is moving files out of an FTP server, the highlighted selection of files on the server needs to be deleted after the transfer after verifying that all files have been transferred successfully, meaning no new entries in the "Failed transfers" list. To get a peace of mind that the selection was transferred successfully, try downloading it again while skipping existing local files. If no new files are downloaded, this ensures all files have already been transferred. This might apply to other software as well.

FTP server applications for mobile devices may handle file listing differently. Some do not report the year of the file, only day and month, causing the FTP client to insert the current year for files except those last modified at a later time of the year than currently, for which the previous year is inserted instead. Another distinction between FTP server apps is whether they list file and directory names starting with a dot, which is considered ''hidden'' in the Unixverse (i.e. on Unix and Linux-based operating systems, which includes Android OS, the most popular mobile operating system).

FTP server applications typically allow the user to select a specific directory to share, rather than the entire storage. This feature has been [[:File:Android extended MTP options concept.png|proposed for MTP]], but never implemented there so far.

Two open-source FTP server apps for Android OS are the integrated FTP server of "Amaze File Manager", and the more sophisticated "primitive FTPd", only the latter of which reports files and folders with names starting with a dot.

Alternatively, files may be uploaded vice versa from the mobile device to an ethernet FTP site served by a home computer, though as of 2021, no mobile file manager's FTP client supports [[File management#Time stamp preservation|preserving files' date and time stamps]] upon uploading.

==== Handling invalid file names ====
Since the most popular mobile operating system, Android OS, is Linux-based, it supports characters in file names that are unsupported by the most popular desktop operating system, Windows, and by some file systems. These characters include a colon (<code>:</code>), a back slash (<code>\</code>), a vertical pipe (<code>|</code>), a question mark (<code>?</code>), and an asterisk (<code>*</code>). Additionally, file names are case-sensitive, meaning files named "file" and "File" and "FILE" can co-exist within a directory on Linux and Android OS, but not on Windows.

While some mobile phone apps disallow the creation of such files and automatically replace characters Windows considers invalid with a substitute character such as an underscore (<code>_</code>), other apps might have created files with names containing aforementioned characters.

When copying or moving such files from a mobile device through MTP (Media Transfer Protocol) using Windows, all characters before and including the last invalid character are discarded from the file name on the target location to make the file openable. Therefore directories on a mobile device which contain such files will need to be moved out in two passes to retain the file names. First, an [[:w:archive file|archive file]] such as a ZIP file should be created on the mobile device which can then be moved out before the files themselves can be moved out.

The archive file can optionally be created after isolating files with invalid characters in a separate directory to consume less space, though that can be a difficult task on a mobile device due to limited file management software and users' infamiliarity with the Linux terminal which can be accessed on Android OS through third-party applications such as Termux or Jack Palevich's "Android Terminal" app.

Another option would be to only create the archive file and not transfer the bare files to the archival media. However, this requires files to be extracted before being opened, which adds a delay for larger files and does not allow for preview thumbails without extracting the entire archive file.

=== On-device management ===
Additionally, file access on the most popular mobile operating system has been restricted significantly over time, and to varying degrees per storage type (internal, memory card, and USB-OTG).

Such restrictions affect third-party applications installed by the user, including file managers. Pre-installed file managers are usually unaffected, though these tend to be functionally restricted, such as lacking range selection, where only two entries need to be tapped for all inbetween to be marked.

[[:File:Android extended storage options concept.png|Options to deactivate these restrictions at user discretion]] were not officially provided, leaving so-called ''rooting'' as the only possibility of bypassing them. This is a process in which the user unlocks administrative access over the operating system.

The operating system vendor claims aforementioned file access restrictions to serve user security, though them being a cloud storage vendor as well suggests a commercial interest that conflicts with end users' desire of freedom, and simultaneously may encourage users to unlock ''root'' access, which is against vendors' recommendation, and where inexperienced tampering can lead to malfunction.

== Other ideas ==
=== Archival queue ===
New files from portable devices which are currently unneeded can be moved into a buffer directory of files ready for archival, which means moving them to a large and stationary hard drive at the next connection to the computer.

External flash storage such as USB sticks and solid state drives can also be used to store data for, for example, the duration of a trip or vacation, where they can be moved to an archive hard drive when arriving at home.

=== Temporary redundant retention after archival ===
Files that have already been moved to a larger stationary archive drive may be redundantly kept on the smaller portable data storage such as a mobile phone or USB stick, but in a directory in which any file is eligible for deletion, such as at space exhaustion.

This would serve as a short-term backup, which could be retrieved from in case anything goes wrong with the archive drive prior to it getting backed up itself.

This increases file fragmentation on the portable device, though that does not noticably affect performance on flash storage.

=== Partition for small files ===
If your computer setup has no secondary drive and/or partition, you may create a small partition (e.g. 4 GB) with a low cluster size for more efficient storage of small files.

Additionally, if space storage happens to be exhausted on the main partition, with software arbitrarily attempting to write to it, files can still be added on the secondary partition without interference.

=== Sensitive environment ===
Inside sensitive environment, data may be exchanged through rewritable optical media such as DVD±RW and BD-RE, as these use external storage controllers, making the media itself unable to contain malicious hardware such as so-called ''rubber duckies'' used to simulate keystrokes from a USB keyboard.

Additionally, finalized write-once media and/or read-only (''ROM'') optical drives can ensure write protection where necessary, for example in a malware-infested environment.

=== Copies of description files ===
Description files such as text files describing the contents of a folder may be stored both inside a folder and a copy in a central location with other description files for easier discovery. The name of that file should contain the name of the folder it is describing.

It is recommended to store such a description file inside the folder it is describing rather than along with it in the parent directory, to make it easier to find if the items in the parent directory are not sorted alphabetically, since a file managers might separate files and folders regardless of which sorting method (alphabetical, by size, by name, by last modified) is chosen, which would make the description file not appear next to the folder it is describing.

== Observations and tips ==

=== Spare directories ===
When space on a device or partition is exhausted, no new directories that could be helpful for organizing files can be created, such as for moving files from a highly ''populated'' directory (i.e. with many files, such as a download folder) on a mobile phone in order to skip having to open the populated folder directly through [[#Media Transfer Protocol|MTP (Media Transfer Protocol)]], which notoriously handles long file listings poorly.

Prepare for such a situation by creating a reserve of spare empty directories inside one dedicated directory. The spare directories can be moved out of the reserve, and be renamed as necessary, even without space left, which allows organizing files on the go to be able to move them elsewhere immediately when arriving home.

=== File move behaviour ===
When the aim is to bring files to an archive, moving files rather than copying and deleting afterwards has the convenience benefit of acting like a check list of files, instead of creating duplicates that would later have to be sorted out without accidentally deleting non-copied files, as well as imminently clearing (freeing) space on the source device.

Moving instead of copying and then deleting files also defeats the psychological barrier that may come from the deletion step, as it feels like a destructive action even though it leads to the same result as moving between storage devices. Another barrier is the uncertainty of having inadvertently selected any file not copied.

When moving files, some file managers may delete files individually after transfer, while others only delete selected files from source only after the last file has finished transfering.

Windows Explorer uses the former method for mass storage devices, but the latter with Media Transfer Protocol. The Linux file manager ''Nemo'' always uses the former method.

With the latter method, any interruption would cancel the file transfer without having freed up any space on the source device.

=== Escape auto-closure ===
Some file managers such as Windows Explorer close themselves when detecting the removal (unmounting and/or physical unplugging) of a storage device, while others jump to the starting directory. Some file managers might do nothing.

The first case may be perceived as annoying, as it forces users to re-open the file manager and navigate all the way back to the previous directory.<ref>[https://www.sevenforums.com/general-discussion/148620-how-stop-windows-explorer-closing-removable-disks.html ''How to stop Windows Explorer closing for removable Disks!'' – SevenForums]</ref>

This can be prevented by opening a different device in the file manager before unplugging. After plugging the device back in, the previously opened directory can be navigated back to immediately through the navigation history, using the on-screen {{button|←}} button or <kbd>Alt</kbd>+<kbd>←</kbd> on the keyboard.

=== Using the command line ===
Command-line file operations can be logged for later reference by using the "verbose" switch <code>--verbose</code> or shorthand <code>-v</code> on the <code>cp</code> and <code>mv</code> commands in Linux and redirecting the output into a text file by appending <code>>>path/to/textfile.txt</code>. On Windows, file operations are outputted by default, meaning no "verbose" switch is necessary. On Linux, using <code>| tee -a path/to/textfile.txt</code> allows visibly printing out command line output in real-time (as usual) while logging into a text file simultaneously.

Autocompletion using the <kbd>↹Tab</kbd> key is widely supported on both Microsoft Windows and Linux/Unix-based operating systems and facilitates navigation by selecting file and folder names.<ref>[https://www.computerhope.com/tips/tip176.htm Use tab to autocomplete commands in the command line (ComputerHope, 2020-12-31)]</ref>

=== Shortcuts ===
Don't hesitate to use bookmarks to frequently accessed directories in your file manager and file picker dialogue (also known as "Open" and "Save as"). If they become unnecessary, they can be removed at any time.

For command-line use, familiarize yourself with environment variables, as they allow quicker navigation throughout your directories. They include <code>~</code> on Linux and <code>%user profile</code> (user home directory, equivalent to <code>C:\Users\''Username''\</code>) and <code>%temp%</code> (<code>C:\Users\''Username''\AppData\Local\temp\</code>).

Environment variables are also typically recognized by file pickers.

Additional shortcuts for command-line use can be specified as variables in the script that runs when starting the terminal, which in Linux is typically located at <code>~/.bashrc</code>. For example, <code>$dl</code> can be set to refer to the download folder by appending <code>dl=~/Downloads</code> to that file.

=== Spaces in file names ===
Keep in mind that space characters in file and path names can become a nuisance when entering paths, creating variables, selecting text, and navigating and auto-completing in a command prompt/terminal using the <kbd>↹Tab</kbd> key.

=== Cleanup ===
"Cleanup" may refer to moving files scattered around the storage and desktop in folders to improve overview, or deleting files like duplicates to clear free space. In the former case, refer to [[/Directory structures]] and consider moving files you are unsure where to move into a [[#Dumping_ground|dumping ground]]. In the latter case, as with [[Backup#lostBackup|lost backups]], first weigh effort against price, meaning here first consider whether the time and effort spent searching files to clean up really outweighs the storage space price.

It is not worth trying to search and eliminate duplicate files for the sake of it if no significant space is cleared. The time and effort necessary to do it might outweigh any benefit from the saved space. However, duplicates may be deleted for organizational purposes, such as duplicate music tracks in the same folder.

Note that cleaning does not necessarily speed up the computer noticeably, except if the partition was nearly full, which should be avoided anyway, as described in [[#Avoid_exhausting_the_operating_system's_partition|§ Avoid exhausting the operating system's partition]]. Rather, closing tasks that stress the CPU and take much RAM is effective in speeding up the system. Clearing caches is mostly unnecessary and even disadvantageous, as they provide faster recall of information.

For files vulnerable to damage like PNG and TIFF images, and [[Backup#Compressed_archives|compressed archives]], duplicates are beneficial for files not backed up yet.

== Experiences by participants ==
This section goes deeper into the idea about that "the average computer and mobile phone user still struggles to keep track of files in the long term despite of all the tools at their disposal". Here participants or visitors to this learning resource can add why they need a resource such as this to learn from and/or interact with it.
:''Moved to [[Talk:{{PAGENAME}}]].''

== References ==
<references />

== See also ==
* [[File naming]]
* [[File systems]]
* [[Backup]]
* [[Data recovery]]

== External links ==
* Comics:
** [https://xkcd.com/949/ File Transfer – xkcd #949, 2011] ([https://www.explainxkcd.com/wiki/index.php/949:_File_Transfer explainer])
** [https://xkcd.com/1360/ Old Files – xkcd #1360, 2014] ([https://www.explainxkcd.com/wiki/index.php/1360:_Old_Files explainer])
** [https://xkcd.com/1817/ Incognito Mode – xkcd #1817, 2017] ([https://www.explainxkcd.com/wiki/index.php/1817:_Incognito_Mode explainer])

[[Category:Information technology]]
[[Category:Computer_science]]

Bash programming/Function Usage

2024-12-13T09:29:22Z

Elominius: /* Examples */ + 2 functions

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Count the files in a directory and its subdirectories: <code>filecount-total() { find "$@" |wc -l; }</code>
* Count the folders only: <code>subfoldercount() { find "$@" -type d |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

* Add time stamps at the beginning of specified file names:
<syntaxhighlight lang=sh>
prepend_timestamps() {
if [[ "$@" == "" ]]; then echo "No file name specified. Exiting."; return 1; fi;
for filename in "$@"; do
timestamp="$(date -r "$filename" +%Y-%m-%dT%H-%M-%S)"
mv -nv -- "$filename" "$timestamp $filename";
done;
}
alias addtimestamps=prepend_timestamps;
</syntaxhighlight>

* Generate check sums of all files in the current or a specified directory:
<syntaxhighlight lang=sh>
superMD5() { find "$@" -type f |sort |xargs -d '\n' md5sum; }
</syntaxhighlight>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>

This works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-11-18T17:57:17Z

Elominius: /* Examples */ Fixed syntax error. This is bash, not JavaScript.

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

* Add time stamps at the beginning of specified file names:
<syntaxhighlight lang=sh>
prepend_timestamps() {
if [[ "$@" == "" ]]; then echo "No file name specified. Exiting."; return 1; fi;
for filename in "$@"; do
timestamp="$(date -r "$filename" +%Y-%m-%dT%H-%M-%S)"
mv -nv -- "$filename" "$timestamp $filename";
done;
}
alias addtimestamps=prepend_timestamps;
</syntaxhighlight>

* Generate check sums of all files in the current or a specified directory:
<syntaxhighlight lang=sh>
superMD5() { find "$@" -type f |sort |xargs -d '\n' md5sum; }
</syntaxhighlight>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>

This works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-11-06T23:55:19Z

Elominius: /* Examples */ unnecessary blank lines

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

* Add time stamps at the beginning of specified file names:
<syntaxhighlight lang=sh>
prepend_timestamps() {
if [[ "$@" == "" ]]; echo "No file name specified. Exiting."; return 1;
for filename in "$@"; do
timestamp="$(date -r "$filename" +%Y-%m-%dT%H-%M-%S)"
mv -- "$filename" "$timestamp $filename";
done;
}
</syntaxhighlight>

* Generate check sums of all files in the current or a specified directory:
<syntaxhighlight lang=sh>
superMD5() { find "$@" -type f |sort |xargs -d '\n' md5sum; }
</syntaxhighlight>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>

This works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-11-06T23:54:05Z

Elominius: /* Examples */ moved to correct section

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

* Add time stamps at the beginning of specified file names:
<syntaxhighlight lang=sh>
prepend_timestamps() {
if [[ "$@" == "" ]]; echo "No file name specified. Exiting."; return 1;
for filename in "$@"; do
timestamp="$(date -r "$filename" +%Y-%m-%dT%H-%M-%S)"
mv -- "$filename" "$timestamp $filename";
done;
}
</syntaxhighlight>

* Generate check sums of all files in the current or a specified directory:
<syntaxhighlight lang=sh>
superMD5() { find "$@" -type f |sort |xargs -d '\n' md5sum; }
</syntaxhighlight>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>

This works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-11-06T23:53:06Z

Elominius: /* Examples */ prepend_timestamps, superMD5

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

** Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>
** Works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

* Add time stamps at the beginning of specified file names:
<syntaxhighlight lang=sh>
prepend_timestamps() {
if [[ "$@" == "" ]]; echo "No file name specified. Exiting."; return 1;
for filename in "$@"; do
timestamp="$(date -r "$filename" +%Y-%m-%dT%H-%M-%S)"
mv -- "$filename" "$timestamp $filename";
done;
}
</syntaxhighlight>

* Generate check sums of all files in the current or a specified directory:
<syntaxhighlight lang=sh>
superMD5() { find "$@" -type f |sort |xargs -d '\n' md5sum; }
</syntaxhighlight>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-10-22T16:38:26Z

Elominius: /* Working with multimedia */ syntax; details

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

=== Working with multimedia ===
* Create a file list for the concatenation feature of <code>ffmpeg</code> from the <code>find</code> command:
** <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fffile >>example.txt </code>
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
** Example use: <code>ffconcat example.txt example.mp4</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

** Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>
** Works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-10-22T16:34:27Z

Elominius: /* Working with multimedia */ Get the resolution of a video

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>getfps() { ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"; }</code>
* Get the resolution of a video: <code>getRes() { echo $(ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=height "$1"; )p; }</ref>
* Create a file list for <code>ffmpeg</code> from the <code>find</code> command: <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fflist</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

** Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>
** Works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2024-10-22T16:25:57Z

Elominius: /* Examples */ short functions fit in a single line; added more functions

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory: <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory: <code>findfile() { find "$2" |grep -i "$1"; }</code>
* Search text inside 7z (7-Zip) archives: <code>7zgrep() { 7z e -so "$2" |grep -i "$1"; }</code>
* Find the newest or the oldest file in a given directory:
** <code>newestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |tail -n 1; }</code>
** <code>oldestfile() { find "$@" -type f -printf '%T+ %p\n' |sort |head -n 1; }</code>

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will overwrite the specified video files in-place, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:a copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"</code>
* Create a file list for <code>ffmpeg</code> from the <code>find</code> command: <code>fffile() { sed -r "s/(.*)/file '\1'/g"; }</code>
** Example use with pipe: <code>find DCIM/Camera/VID_{{#time:Ymd}}*.mp4 |fflist</code>
* Stick all videos from one folder into one video file without re-encoding (only works for files from the same device with the same width and height and frame rate):
<syntaxhighlight lang=sh>
ffconcat-dir() {
# edge case handlers
if [[ "$1" == "" ]]; then echo "No directory specified. Exiting." ; return 1; fi
if [[ "$1" == "ffconcat-dir.txt" ]]; then echo "This name is reserved. Please choose a different directory."; return 1; fi
if [ -d "ffconcat-dir.txt" ]; then mv ffconcat-dir.txt "ffconcat-dir (usurped-$(date +%Y%m%d%H%M%S))"; fi

# main part
truncate -s 0 ffconcat-dir.txt # blanking temporary file from last run
find "$1" -maxdepth 1 -type f |sed -r "s/(.*)/file '\1'/g" >ffconcat-dir.txt
if [[ "$2" != "" ]]; then outfile="$2"; else outfile=tmp.mp4;fi
ffmpeg -f concat -safe 0 -i ffconcat-dir.txt -c copy "$outfile"
}
</syntaxhighlight>

** Example use: <code>ffconcat-dir folder_name output_video_name.mp4</code>
** Works for other file types too, but not with mixed file types. The extension specified file type has to match the type of the source files. The default name is <code>tmp.mp4</code> if none is specified.

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

File management

2024-10-22T16:03:27Z

Elominius: /* Deduplication */ A guide to deduplication.

This lesson teaches the management of digital files and directories.

While there exists no ideal method of file management, this resource documents possibly helpful practices and inspiring ideas.

This lesson was created under the impression that the average computer and mobile phone user still struggles to keep track of files in the long term despite of all the tools at their disposal.

== Descriptive file names ==
Every file manager has a "rename" feature, letting you change the names of files and folders.

Don't just let this essential feature "rust in the junk yard". Utilize it to make files easily searchable. For example, add a short description after a picture or video file name. If you recorded using a [[smartphone]], you can rename the file on the device itself right afterwards. For example,

* <code>VID_{{#time:Ymd_His}} trampoline jumping.mp4</code>

It is recommended to add the short description after the file name rather than before, to prevent interference with alphanumerical sorting. Even though file managers can still sort items by date and time, files' time stamp attribute might not always be retained throughout transfers, for example when transferring files through [[:w:Media Transfer Protocol|MTP]], between a smartphone's internal memory and memory card, or uploading files to a cloud storage service. A time stamp at the beginning of the file name allows alphanumerical sorting to act as chronological sorting even if the date and time attribute is lost.

Not every single file needs to be given a name manually, since that would take much effort. If your file manager lacks range selection and bulk renaming, meaning the selection and renaming of many items at once, adding a descriptive file name to the first file of similar files will still facilitate finding those files, since the title of the first file in the group covers the remaining files. For example, all four of these imaginary files depict strawberries, but only the first was manually named:

* <code>IMG_20230914_090112 strawberries.jpg</code>
* <code>IMG_20230914_090116.jpg</code>
* <code>IMG_20230914_090119.jpg</code>
* <code>IMG_20230914_090124.jpg</code>

== Directories ==
Make use of directories. Categorize files into logically structured directories to facilitate finding them later.

Some inexperienced computer users might have a habit of just throwing files into the root directory of their device such as a flash drive. With an increasing count of accumulating files, they would become increasingly difficult to find, especially multimedia content which can not simply be searched for text strings using tools like `grep` but only through metadata.

Create a mind map by asking yourself where you would look for these files whenever necessary.

Below are two example directory structures for video projects, where the project files of the first project are directly in the project folder, while those of the second are in a subfolder; a matter of preference.

<blockquote>
* Video projects
** Example project 1
*** Assets
** Example project 2
*** Project files
*** Assets
</blockquote>

Structure files in the order your mind recalls them. For example, the type and source of files is more memorable than the date. For more examples of directory structures, see the [[/directory structures]] sub page. Feel free to add examples to it.

For directories where a sufficiently descriptive directory name would be considered too long, consider creating a descriptive text file inside named <code>description.txt</code>, <code>info.txt</code>, or similar. Comments about specific files may be noted in a text file with <code>.meta</code>, <code>.meta.txt</code>, or a similar suffix appended to its name.

If files within the same category have wildly varying sizes, such as command line outputs, you might want to move files beyond a size threshold such as 1 MB or 5 MB into a separate folder whose name can be the same with an added "-large" suffix. This allows minimizing the size of compressed snapshots of the directory.

Avoid putting sub-directories in directories populated with many files, since they can become a nuisance when wanting to view the lastest file whereas the file manager lists sub-directories at the top, or otherwise be difficult to find between the files when needed. Due to the slow loading times of the [[:w:Media Transfer Protocol|Media Transfer Protocol (MTP)]] which is commonly used to connect mobile devices to desktop/laptop computers, and due to the possibility of file managers trying to determine file types by reading internal file information,<ref>''[https://github.com/linuxmint/nemo/issues/1907 nemo significantly slower opening folder content #1907 - GitHub]'' ([[:w:Nemo (file manager)|Nemo file manager]] reads the [[:w:Header_(computing)|header]] of each file with unknown [[:w:Filename extension|extension]]</ref> it could make navigating more difficult. Therefore, it should be considered to create a separate folder that only contains subfolders, such as <code>Downloads_2</code>.

You don't need to spend much effort coming up with good names for files and folders, as you can change them later at any time if you happen to come up with a better idea.

== Revision history ==
When spending much time on a project, write-up, email draft, etc., consider saving a revision history, by selecting "Save as" (also accessible through <kbd>Ctrl</kbd>+<kbd>↑Shift</kbd>+<kbd>S</kbd> keyboard shortcut on some software) and changing the file name through numbering or timestamping, or alternatively creating renamed copies externally using a file manager or command terminal. For possible file naming schemes and variations of numbering and time-stamping, see [[File naming]].

This enables you to revert to an earlier revsion in case of error, prevents total loss in case of failed writes caused by a power outage or software crashing, and later facilitates comprehending the work progression, while only consuming marginial space compared to common data storage, and being efficiently [[Backup#Compressed_archives|compressible]] due to redundancy.

A new revision does not necessarily have to be created upon every saving, but whenever the changes since the last revision are major enough at your discretion. Optionally, a short comment summarizing the changes can be added to the file name.

For programming code, suffixes like <code>-stable</code> and <code>-unstable</code> may be added after the time stamp, for example: <code>UserScriptName-revision-{{#time:YmdHis}}-stable.js</code>. Separating the number or time stamp allows convenient double-click selection in the file saving dialogue.

Revisions can optionally routinely be moved into a separate subfolder.

== Dumping ground ==
For files and folders you are unsure where to put, consider creating a directory on your device named <code>dump</code>, <code>sandbox</code>, or similar.

You may wish to categorize those into text, compressed archives, drawings, or by whichever task, or dated [[file naming|folder names]] such as <code>{{#time:Y-m-d}}</code>.

When managing files and directories in a command line terminal, an item can temporarily be given a simple and short name such as <code>1</code> to facilitate typing in the commands, that will be changed shortly after. Examples are:

; Output from multiple commands into one log file:
<syntaxhighlight lang=sh>
$ ls [path] -alR >>1 # Write names and attributes of that folder's content into the file "1".
$ find [path] >>1 # Write a list of bare file paths into the file "1".
$ mv -n 1 [desired file name] # Rename "1" to desired file name. "-n" prevents accidental overwriting.
</syntaxhighlight>

The above command could also be done in a single line with <code>(ls ''[path]'' -alR; find ''[path]'') >>''[desired file name]''</code>, but the former might be preferred to save time if the target directory contains a high number of files, as the former command starts immediately after pressing <kbd>⏎ Return</kbd> (also known as "Enter"), or when one wishes to enter commands before thinking of an output file name, while the latter requires typing in the whole command first.

; Moving files from multiple sources into a directory:
<syntaxhighlight lang=sh>
$ mkdir 1 # Create a folder named "1".
$ mv -n *.mp4 *.mkv 1 # Move mp4 and mkv files into it.
$ mv -n 1 Videos-$(date +%Y-%m-%d) # Renaming folder to intended name.
</syntaxhighlight>

Should it be necessary to print out the absolute (full) file path of a file for, for example, quick copying to the clipboard, use the command <code>readlink -f ''[target file]''</code>, and for information about the current mount point, use the <code>findmnt -T .</code> command.

== Finished tasks ==
Files whose task has finished, such as a posted message that was initially drafted offline into a text file, can be moved into a subfolder of the current directory named <code>done</code>, or alternatively in one big shared folder for this purpose.

== Storage types ==
Auxiliary data which is frequently accessed can reside on the operating system drive, on its own partition or a separate one. This especially applies to portable computers and operating systems installed on external media such as a USB flash drive.

=== Secondary and external storage ===
On desktop computers, large files can reside on a secondary large hard disk drive or solid state drive, the former of which costs less per space. For laptops, stationary hard drives located at home or portable external hard drives or solid state drives can be used. While working with a laptop as passenger in any moving vehicle such as a bus or tram, solid state memory is preferrable due to sturdiness, as hard disk drives do not like constant physical movement due to mechanical wear. If portable storage necessary, files can be stored on a constantly inserted memory card, which does not compromise ergonomy as they don't physically protrude, or at most a little. In addition, it can be occasionally removed when the data is needed on a different device. SD cards with 1 TB exist since at least 2017, though are expensive.<ref>[https://www.engadget.com/2016-09-20-sandisk-1tb-sd-card.html ''SanDisk outs the 'world's first' 1TB SD card''] (2016-09-20)</ref>

A home server adds the convenient benefit of access from all devices at home, and even through the internet if set up by the user ("private cloud"), but typically has longer latency times (access delays) than physically attached storage, and lacks mass storage access that may be necessary for some programs to work properly. In such a case, the files would have to be downloaded first and worked with locally, and uploaded after finishing.

=== Incubate work on flash storage ===
If, for example, your computer has a setup with solid state drive for operating system and hard disk as expansion storage, a project may be worked on on the flash storage, and changes can be applied to the hard drive at the end of a day.

If the hard drive is set to spin with no or a long timeout such as an hour or more, this may not be as necessary, but for short timeouts, frequent spin-ups would cause mechanical wear and tear in addition to annoying delays.

Using flash memory can be particularly useful on battery-powered laptops for power efficiency, though mechanical hard drives are increasingly being usurped by solid state memory in laptops, mainly due to physical robustness, as hard drives on laptops have long served for cost saving, whereas solid state memory is becoming increasingly affordable since the mid-2010s, though external USB hard drives may be used on the go.

=== Hard drives' purposes ===
After purchasing a hard drive, choose whether to dedicate it to either auxiliary storage or archival ([[wikipedia: cold data|cold storage]]).

Use as auxiliary storage such as for a workstation or server demands the drive be in constant operation, which wears it down over time, making it unsuitable for long-term archival.

Use as archival storage demands only sporadic (rare) operation to add or retrieve data, which induces far less mechanical wear.

Data retrieved from an archive drive should be copied over to auxiliary storage to avoid needless wear on the former whenever that data is recalled again in near future, as can be expected, and in order to further duplicate files that have proven to be useful, as archival is commonly done with an uncertainty of which files will be useful in future. A functional archive drive may later be repurposed for auxiliary storage after moving the archived data over to a new device.

Make a list of data you wish to retrieve the next time you access your archive drive. If you can afford it, mirror the archive to a readily available hard disk or network-attached storage. If your live hard disk or network attached storage runs out of disk space, delete the files that were not accessed for the longest time if they are already backed up somewhere else. To find them, file managers are equipped with the ability to sort files and directories by "last accessed" (or similar) in either direction.

=== Labels ===
Consider physically labeling your storage media to facilitate finding data. For example, a date or a short summary of the contents can be written on a label sticker that is applied onto the casing of an external data storage device.

For optical discs, a disc marker (also interchangeably known as "CD marker", "DVD marker", etc.) can be used to write notes directly on the disc. If more space is necessary, label stickers on its containing [[:w:Optical_disc_packaging#Jewel_case|jewel case]] or spindle can be used. Many jewel cases include a paper sheet for notes. Do not attach stickers directly onto the disc, as they could disintegrate during high rotation speeds, risking damage to the optical drive's internal components.

It is also recommended to change the device's file system label to a year or a short summary of the contents so it can quickly be seen in the file manager's device list (usually a side bar on the left) when plugged in.

On data storage devices whose life expectancy appears to be nearing its end, as indicated by [[:w:Self-Monitoring,_Analysis_and_Reporting_Technology|S.M.A.R.T. data]] on hard drives and performance loss on solid state memory. an "expired" or "EOL" (end of life) label can be added. Such devices should at most be used for temporary purposes such as testing.

== Data storage size ==
It is recommended to get a larger storage capacity than one intends to use. This leaves some spare room in the case that more data than expected is created, such as vacation video recordings.

On portable devices such as mobile phones and digital cameras, a larger storage capacity lowers the required frequency of file transfers into ones archive.

Another benefit of larger storage media is that more data can be readily accessible at a given time, and a needed file can be found faster since fewer devices have to be searched. However, a central file list and [[#Labels|labeling]], as described in other sections, can facilitate finding files as well.

If your storage device has, for example, 128GB, it realistically means 100 GB. Not only because of reserved space from file system overhead and possibly operating system files, but because of those 128 GB, the end is less useful, since one might want to put files there that are slightly larger than the remaining space, and one would like to have all those files in one location to avoid confusion and the annoyance of changing media.

In other words, if one would like to put 11 GB where only 10 GB are available, those 10 GB are as little useful as 0 GB.

== Partitioning ==
User data may be stored on the same or a separate partition as the operating system.

The benefit of a separate partition for user data is that possible file system corruption on the operating system partition would not spill over to user data, though modern file systems such as ''NTFS'' and ''ext4'' protect themselves from damage by [[:w:Journaling file system|journaling]], which allows the file system driver to recover quickly after an unexpected termination of write access caused by an operating system crash or unexpected removal.
Infrequently accessed files that may be necessary in near future can be stored there as well, or moved to a secondary drive if their size is significant.

Another benefit of a separate user data partition is the smaller backup size of the operating system partition and facilitated recovery in case of a malfunctioning operating system where other means of repair have failed or would be too difficult. Because operating systems are subject to corruption and can at worst become unbootable, it is good practice to [[backup|back up]] their partition regularly into a disk image. A smaller operating system partition can be imaged more quickly and the routine induces less wear on the backup media. Should the operating system malfunction, it can be imaged and then restored more quickly from the functional previous disk image, with less work to merge desirable changes since the last backup.

== Packing and archival ==
Perpetual streams of new files such as web downloads, photos and videos from digital cameras and cell phones, and screen captures can be packed by renaming their parent folder into a uniquely identifying name, such as with date stamp: <code>Camera-{{#time:Y-m-d}}</code>, after which they can be moved to an archive drive at the next backup appointment. The folder's name may also contain a location, device type, and/or short description. If packed more than once on the same day, time stamps or part numbers can be added to the names, or the files can be merged into the same directory.

If you no longer intend to change the contents of an archived folder, a file count and byte size can optionally be added to the name to provide a quick overview in file lists and to facilitate verifying whether all data has been transferred without having to navigate back to the source device and wait for loading to view the folder's byte size. An example name is <code>Camera-{{#time:Y_m_d}}-12147367878b-2093items</code>. It is recommended to use the exact byte number to prevent confusion between file size units that are powers of two (KiB, MiB, GiB) and powers of ten (KB, MB, GB).

Alternatively to renaming on the source device, the directory with uniquely identifying name can be created on the archive drive first, and files can be moved there out of the source directory.

When to pack files is end users' decision, though it is recommended to do so before exhausting free space on the source device. Renaming is not as necessary if new folders are created automatically, like [[:w:Digital_camera#Directory_and_file_structure|some digital camera firmwares do per 999 or 1000 pictures]]. All filled folders can be considered eligible for archival.

Larger storage space provides the benefit of more buffer until the next file transfer becomes necessary to clear space, thus it needs to be done less frequently.

Individual files needed for a specific purpose such as an impending project can be copied or moved into a separate directory.

== Write protection ==
Write protection may be desirable to defeat the fear of accidental modification of data when not desired by the user.

A simple way of achieving write protection in Linux-based operating systems is to mount or re-mount a device or partition as read-only with this command which requires superuser privileges: <code>mount -o remount,ro ''[device or mountpoint]''</code>.

If write-protection is not supported by the operating system, an SD card with write protection switch feature can be used. The switch relies on the SD card reader to obey it and deny writing access to the operating system. Some memory card readers, both built-in ones and USB adapters, might not obey the write protection switch.

Another way to achieve write protection is finalized write-once optical media or a read-only optical drive with insufficient laser beam power to write data, as described in {{section link||Sensitive environment}}.

== File listing ==
Searches within file lists inside a text file are significantly faster than searches through a file system.

See [[Data_recovery#Create_a_file_list|this guide]] on how to create file lists.

=== File index ===
As explained in {{section link|File_puzzling#Orphaned_directories}}, some file systems store directories that comprise file paths as "linked lists", meaning distributed over the entire space rather than one index of "nodes" at the beginning, which has both benefits and disadvantages, the latter of which is slower file searching.

A searchable file index stored in a text file named "index" created using the <code>find >index</code> command can facilitate finding files, as it contains a list of paths to all files at one place. The index can be updated by running the command again to overwrite the existing one. If the working directory is not the root of the file system, it should be changed to it or the paths need to be specified.

The "index" file can be searched with ease using <code>grep -i "searched file name" index</code>, which is typically much faster than directly searching the file system. <code>-i</code> may be left out for case-sensitive search. These commands have additional options, but these are outside the scope of this section.

== Time stamp preservation ==
Some methods of file transfer, such as copying within/onto mobile phone storage, the <code>cp</code> command without activated <code>-p</code> ("preserve") option, and a directory on Unix/Linux not owned by the current user, might discard date and time stamp file attribute(s), resetting it to the current time.

To preserve last-modified time stamps over FTP, downloading is preferred, as uploading while preserving it requires both client and server to support the [[:w:FTP#Additional_commands|MDTM (''Modify Fact: Modification Time'') command]], which it is not widely.

== High numbers of small files ==
With an increasing number of files, file searches slow down. High numbers of small files also restrict portability, as they demand more file operations for file transfers, slowing the process down.<ref>Related: [https://www.quora.com/Why-is-copying-1-000-1MB-files-so-much-slower-than-copying-1-1GB-file-given-that-the-same-amount-of-data-is-being-copied/answer/Franklin-Veaux Answer on Quora to "''Why is copying 1,000 1MB files so much slower than copying 1 1GB file, given that the same amount of data is being copied?''" by Franklin Veaux, on January 14th, 2020]</ref> Additionally, higher cluster sizes in combination waste more space to [[File Systems#Cluster_size|cluster overhead]] (unused reserved space).

If you happen to have a high number of currently unneeded small files, such as tens of thousands, consider packing them into one big archive file for improved portability.

Compression may be considered where efficient, such as in human-readable text files and code, and/or where more necessary, such as online file sharing. Compression ratios of 100 may be achievable by strong compression algorithms on text documents and code. However, it should be taken into consideration that damage magnifies enormously over compressed archives, as demonstrated in {{section link|Backup#Compressed archives}}. Therefore, it is recommended to store compressed archives on at least two devices.

Text inside compressed archives can be searched through directly without extracting using tools such as <code>zgrep</code>, <code>xzgrep</code>, <code>bzgrep</code>, and for 7-Zip, <code>7z e -so -bd ''[path]'' |grep ''[query]''</code>.

If frequent modification of small files is necessary, an alternative to packed archive files (such as <code>.tar</code>, <code>.zip</code>) are file system containers (''virtual disks'') with a small cluster size. File system containers are manually generated disk image files that can be mounted like usual drives.

== Avoid exhausting the operating system's partition ==
Exhausted space storage should be avoided, especially on an operating system partition, as it could lead to bogus behaviour by software not designed to handle such condition, or other unwanted behaviour. For example, a failed write while saving could blank the target file, causing the loss of work and reset of configuration. A web browser might [[Backup#Browsing_history|automatically delete early browsing history entries]] to make space for new. Even seemingly basic features such as command line parameter completion could malfunction.<ref>{{cite web |url=https://unix.stackexchange.com/questions/277387/tab-completion-errors-bash-cannot-create-temp-file-for-here-document-no-space |date=2016-04-18 |access-date=2022-02-10 |title=Tab completion errors: bash: cannot create temp file for here-document: No space left on device |website=Unix & Linux Stack Exchange }}</ref>

On an operating system partition, keeping a safety margin of free storage such as 5% at any time is recommended, and at least 1% on secondary expansion storage. On archival media, a controlled exhaustion of space is less critical, though the readability of the final written files should be verified.

Should you still find yourself with 100% exhausted space, first seek few megabytes of files to move out, perhaps temporarily, to improve system stability. This allows you to take time to calmly search for more and/or larger files to move out.

== File system repairs ==
Tools such as <code>CHKDSK</code> on Microsoft Windows and <code>fsck</code> on Linux promise repairing damaged file systems. Logical file system errors may be caused by unexpected power outages or unpluggings.

Be careful with file system repairs. It is recommended to back up any device (either to a full-disk image or by copying all files) prior to running one, in order to be able to revert with ease in case of unwanted collateral damage.

Detected file system errors may be caused by incompatibile file names across operating systems. For example, Linux allows characters in file and folder names in NTFS that Windows considers invalid, such as a colon (<code>:</code>) and a pipe character (<code>|</code>), as well as case-insensitive file names. Upon detection of invalid characters, the <code>CHKDSK</code> tool moves and renames such items, which leads to the loss of file names and paths. In particular, <code>ChkDsk</code> moves files and folders with invalid names into a directory located at the file system's root named <code>found.000</code>, and renames them to generic names like <code>file00000000.chk</code> and <code>dir_00000000.chk</code>, where the number is hexadecimal and incremented.

== Disk usage analysis ==
[[File:GNOME_Disk_Usage_Analyzer_3.32_screenshot.png|thumb|Disk usage analyzers facilitate finding directories with the largest content size. Some illustrate the directory structure graphically.]]
Disk usage analyzers calculate the size of directories on any selected path, allowing the user to easily discover directories which occupy the most space. Large folders not currently needed can be moved over to an archive drive, which clears the most space on the source device.

Popular tools for desktop operating systems include ''[[:w:Disk_Usage_Analyzer|Baobab]]'' for Linux (pre-installed on some popular distributions) and ''Xinorbis'' for Windows, both with sophisticated graphical user interfaces. Linux is also equipped with the command-line tool <code>du</code>, which allows outputting results directly into a text file.

For mobile (Android OS), [[:w:ES_File_Explorer|ES File Explorer]] is equipped with such functionality, though that application has been subject to controversy and has developed into adware.
<div style="float:none; clear:both;"></div>

== Deduplication ==
Several tools to automatically deduplicate files exist: rdfind, fdupes, jdupes, rmlint, dupeGuru, FSlint.

Storing one duplicate of files may be desirable in certain situations, such as [[Backup#Compressed_archives|compressed archives]] intended for long-term preservation, where even the slightest damage can render any data after the point of damage unreadable. For accessing the same file from different locations, [[:w:Hard_link|hard link]]s or [[:w:symbolic link|symbolic link]]s can be used.

You may want to deduplicate files across two different storage devices without accidentally deleting any files that do not exist on the storage device you want to keep the files on. A manual deduplication may be necessary if you wish to free up space from a device after copying files to a different device or after creating an archive file from those files without accidentally deleting any files that were not copied or archived from the source device, or after a file copying operation was interrupted and you need to clean up without deleting any files that were not copied.

The quickest way to accomplish this would be to compare the total count and size of those files in both the source and the target device by selecting the files and opening the "Properties" window. If the byte counts match, the source folder can be safely deleted. However, if the selection contains subdirectories too, differences in file systems mean that the total size might not match due to differences in how directories are stored. Some file managers also don't show the exact byte count.

If you want to deduplicate files between two different computers or between a computer and a smartphone, an additional problem is that file managers count files and folders differently. Some file managers only show the number of "items", meaning both files and folders, where as others may only show the number of "files" without counting the folders. If possible, use the same file manager on both devices, and one that shows the exact byte count.

A more sophisticated way to manually deduplicate files is creating a temporary script that contains a list of files to be deleted. In Linux, this can be accomplished first changing the working directory to the target folder (where the files are supposed to stay) using <code>cd</code> and then creating a temporary script by running:

<syntaxhighlight lang=sh>
# list files to be deduplicated
find -type f |sed -r "s/(.*)/rm -v '\1'/g" >>temporary.sh
# list empty directories to be removed
find -type d |sed -r "s/(.*)/rmdir -v '\1'/g" >>temporary.sh
# script deletes itself after work done
echo "rm temporary.sh" >>temporary.sh
</syntaxhighlight>

No files have been deleted up to this point. Only the disposable script to do the work has been created. This script contains a list of files to be deleted from the source device, but the list is generated from the target device to avoid containing the name of any files that have not been copied. It is recommended to use a temporary script rather than throwing around commands involving an <code>|xargs</code> pipe, since running <code>|xargs</code> commands in combination with deletion is dangerous since you don't see a list of files before starting the deduplication process.

The <code>-v</code> flags are optional and serve to later show the names of the files deleted in the deduplication process in the terminal.

Now move the <code>temporary.sh</code> file to the original directory on the source device from which you want to delete the duplicate files. Then change your terminal's working directory to that directory using <code>cd ''[path]''</code>. Before running the script, open it in a text editor and glance over it to verify that it doesn't contain any files you don't intend to delete. Then run it using <code>sh temporary.sh</code>.

Do not run the script before changing your working directory to the same directory the script resides in, because the script contains relative paths rather than full (absolute) paths, so the path of your terminal's working directory will be presumed to be infront of the relative paths in the script.

== Temporary folder ==
Programs may use temporary folders such as <code>/tmp/</code> and <code>~/.cache/</code> on Linux and <code>%temp%</code> on Windows to store data such as preview thumbnails and data from the web to reduce loading times.

Should the way your operating system or file manager handles its [[:w:Temporary folder|temporary]] or [[:w:Trash (computing)|trash folder]] not suit your needs (e.g. if retention span is unchangeable or limited), you may wish to manually operate such in your user home folder (e.g. <code>~/tmp/</code>, <code>~/trash/</code>). Some software may allow changing the path for the temporary folder. Furthermore, such folders can be used for frequent short-term backups of projects, which can be deleted when free space becomes necessary.

In comparison to a traditional "Trash" folder implemented by the operating system or file manager, files can be opened directly as usual, whereas a file manager may disallow directly opening files sitting in a "Trash" folder and show file properties instead, as Windows Explorer does, whereas, for example, the ''Nemo'' file manager for Linux allows directly opening files located in the trash directory.

== Mobile ==
=== Memory cards ===
[[File:Huawei_U8950D_no_cover.JPG|thumb|right|Smartphone with inserted memory card (located below camera lens)]]
Some smartphones and tablet computers allow the expansion of storage capacity using memory cards, typically MicroSD, which significantly facilitates file management and is user-friendly.

Memory cards can be re-used immediately between devices without need for file transferring, and data stored on the memory card is not at the risk of mobile devices' technical defect, as it can be ejected, after which data can be retrieved externally. Mass storage access from an external computer also may allow [[data recovery|recovering some files]] imminently after a deletion accident caused by bogus software<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/ Users report Android devices' entire internal user storage being deleted instantly, caused by poor software design – "''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?''" – Reddit.com/r/Android (2013-01-11)]</ref> and/or human error.

For huge file transfers, ejecting the memory card and directly transferring to the PC through mass storage may save time compared to MTP (media transfer protocol) through the phone or tablet, as the latter does not handle high counts of files within a directory well. Additionally, memory cards can immediately be reused in a different device without lengthy file transfers. USB-OTG (''On The Go'') may be used as well, connected through an adapter directly to the mobile device, though it might not [[#Time_stamp_preservation|preserve a date and time attribute]]. Tablet computers with desktop operating system are widely equipped with at least one default-sized USB-A port.

Additionally, using a memory card takes stress off the device's non-replaceable internal memory, preserving its limited rewrite cycles, which is especially beneficial for repeated heavy tasks such as high-resolution filming and mobile FTP server hosting.
<div style="float:none; clear:both;"></div>

=== Between computer and mobile ===

==== Media Transfer Protocol ====
File management on mobile phones and tablet computers with mobile operating systems is more restricted than on desktop/laptop computers and mass storage devices such as USB sticks and memory cards, as the media transfer protocol (MTP), which is used to access files on a mobile device from a computer, lists files slowly, which is problematic for loading directories with high counts of files.

As such, it is recommended to manage such directories on the device itself. If transfer between a desktop/laptop computer is desired, handle those files in a little to unpopulated directory.

MTP file listing can be sped up by not loading preview thumbnails. Depending on file manager used, this may be done by deactivating preview thumbnails in settings or choosing the "detail" view mode, where files appear in a list instead of a grid which foregoes the loading of preview thumbnails. However, files can be dropped into a directory without opening the target directory, by pasting them through the right-click context menu.

A benefit of MTP is it not being prone to file system corruption as a result of unexpected removal, meaning without being "safely unmounted" through the client operating system (i.e. desktop / laptop computer), as it operates through an abstraction layer and the file system is controlled by a driver on a battery-powered host device (i.e. smartphone / tablet).

Only a selection of files, no directories, should be moved away from the device, because users have reported files on MTP not being listed properly.<ref>[https://android.stackexchange.com/questions/136839/files-show-up-on-nexus-5-but-not-in-windows-7 ''Files show up on Nexus 5 but not in Windows 7'' – Android Stack Exchange – February 11th, 2016]</ref><ref>[https://android.stackexchange.com/questions/46315/not-all-files-are-visible-over-mtp ''Not all files are visible over MTP'' – Android Stack Exchange – May 29th, 2013] (209,502 views as of September 21st, 2022)</ref><ref>[https://stackoverflow.com/questions/13737261/nexus-4-not-showing-files-via-mtp ''Nexus 4 not showing files via MTP'' – StackOverflow – December 6th, 2012] (67,504 views as of September 21st, 2022)</ref> If a directory is moved away from the device, the computer might delete it from the mobile device without all content having been transferred away. Instead, it should be copied, and the byte size be compared on both the computer and the smartphone itself, where a match indicates a successful transfer, meaning the directory can now be deleted from the mobile device. The only exception where moving folders out is safe is when the number of files within is overseeable, i.e. less than ten, where all files are clearly listed in the computer's file manager.

Windows Explorer additionally displays files while listing is in progress, which can be of use when moving files out, since the loading of the file list can be interrupted to allow moving out the displayed files chunk-wise, reducing the number of remaining ones each time.

Transferring files onto the device through MTP [[#Time_stamp_preservation|may dismiss their date and time attribute]].

If a file has newly been created on the smartphone while it was connected to the computer through MTP, the computer's file manager could potentially misreport the file size as too small due to having loaded the directory listing in a moment where the file was not complete. Moving the file away from the phone could cause it to be truncated (incomplete) on the target path while being deleted from the source, since the file manager might wrongfully assume that the file has been fully transferred while it hasn't.

==== File Transfer Protocol ====
An alternative to MTP is FTP (file transfer protocol) through ethernet.

On the desktop computer, a dedicated and sophisticated FTP client such as FileZilla (open-source) may be used to handle high numbers of files, though FTP is widely supported by file managers and web browsers.

FileZilla does not support moving files out of an FTP server, meaning downloading and deleting automatically, whereas moving within a server is supported through the standard rename command. If the intention is moving files out of an FTP server, the highlighted selection of files on the server needs to be deleted after the transfer after verifying that all files have been transferred successfully, meaning no new entries in the "Failed transfers" list. To get a peace of mind that the selection was transferred successfully, try downloading it again while skipping existing local files. If no new files are downloaded, this ensures all files have already been transferred. This might apply to other software as well.

FTP server applications for mobile devices may handle file listing differently. Some do not report the year of the file, only day and month, causing the FTP client to insert the current year for files except those last modified at a later time of the year than currently, for which the previous year is inserted instead. Another distinction between FTP server apps is whether they list file and directory names starting with a dot, which is considered ''hidden'' in the Unixverse (i.e. on Unix and Linux-based operating systems, which includes Android OS, the most popular mobile operating system).

FTP server applications typically allow the user to select a specific directory to share, rather than the entire storage. This feature has been [[:File:Android extended MTP options concept.png|proposed for MTP]], but never implemented there so far.

Two open-source FTP server apps for Android OS are the integrated FTP server of "Amaze File Manager", and the more sophisticated "primitive FTPd", only the latter of which reports files and folders with names starting with a dot.

Alternatively, files may be uploaded vice versa from the mobile device to an ethernet FTP site served by a home computer, though as of 2021, no mobile file manager's FTP client supports [[File management#Time stamp preservation|preserving files' date and time stamps]] upon uploading.

==== Handling invalid file names ====
Since the most popular mobile operating system, Android OS, is Linux-based, it supports characters in file names that are unsupported by the most popular desktop operating system, Windows, and by some file systems. These characters include a colon (<code>:</code>), a back slash (<code>\</code>), a vertical pipe (<code>|</code>), a question mark (<code>?</code>), and an asterisk (<code>*</code>). Additionally, file names are case-sensitive, meaning files named "file" and "File" and "FILE" can co-exist within a directory on Linux and Android OS, but not on Windows.

While some mobile phone apps disallow the creation of such files and automatically replace characters Windows considers invalid with a substitute character such as an underscore (<code>_</code>), other apps might have created files with names containing aforementioned characters.

When copying or moving such files from a mobile device through MTP (Media Transfer Protocol) using Windows, all characters before and including the last invalid character are discarded from the file name on the target location to make the file openable. Therefore directories on a mobile device which contain such files will need to be moved out in two passes to retain the file names. First, an [[:w:archive file|archive file]] such as a ZIP file should be created on the mobile device which can then be moved out before the files themselves can be moved out.

The archive file can optionally be created after isolating files with invalid characters in a separate directory to consume less space, though that can be a difficult task on a mobile device due to limited file management software and users' infamiliarity with the Linux terminal which can be accessed on Android OS through third-party applications such as Termux or Jack Palevich's "Android Terminal" app.

Another option would be to only create the archive file and not transfer the bare files to the archival media. However, this requires files to be extracted before being opened, which adds a delay for larger files and does not allow for preview thumbails without extracting the entire archive file.

=== On-device management ===
Additionally, file access on the most popular mobile operating system has been restricted significantly over time, and to varying degrees per storage type (internal, memory card, and USB-OTG).

Such restrictions affect third-party applications installed by the user, including file managers. Pre-installed file managers are usually unaffected, though these tend to be functionally restricted, such as lacking range selection, where only two entries need to be tapped for all inbetween to be marked.

[[:File:Android extended storage options concept.png|Options to deactivate these restrictions at user discretion]] were not officially provided, leaving so-called ''rooting'' as the only possibility of bypassing them. This is a process in which the user unlocks administrative access over the operating system.

The operating system vendor claims aforementioned file access restrictions to serve user security, though them being a cloud storage vendor as well suggests a commercial interest that conflicts with end users' desire of freedom, and simultaneously may encourage users to unlock ''root'' access, which is against vendors' recommendation, and where inexperienced tampering can lead to malfunction.

== Other ideas ==
=== Archival queue ===
New files from portable devices which are currently unneeded can be moved into a buffer directory of files ready for archival, which means moving them to a large and stationary hard drive at the next connection to the computer.

External flash storage such as USB sticks and solid state drives can also be used to store data for, for example, the duration of a trip or vacation, where they can be moved to an archive hard drive when arriving at home.

=== Temporary redundant retention after archival ===
Files that have already been moved to a larger stationary archive drive may be redundantly kept on the smaller portable data storage such as a mobile phone or USB stick, but in a directory in which any file is eligible for deletion, such as at space exhaustion.

This would serve as a short-term backup, which could be retrieved from in case anything goes wrong with the archive drive prior to it getting backed up itself.

This increases file fragmentation on the portable device, though that does not noticably affect performance on flash storage.

=== Partition for small files ===
If your computer setup has no secondary drive and/or partition, you may create a small partition (e.g. 4 GB) with a low cluster size for more efficient storage of small files.

Additionally, if space storage happens to be exhausted on the main partition, with software arbitrarily attempting to write to it, files can still be added on the secondary partition without interference.

=== Sensitive environment ===
Inside sensitive environment, data may be exchanged through rewritable optical media such as DVD±RW and BD-RE, as these use external storage controllers, making the media itself unable to contain malicious hardware such as so-called ''rubber duckies'' used to simulate keystrokes from a USB keyboard.

Additionally, finalized write-once media and/or read-only (''ROM'') optical drives can ensure write protection where necessary, for example in a malware-infested environment.

=== Copies of description files ===
Description files such as text files describing the contents of a folder may be stored both inside a folder and a copy in a central location with other description files for easier discovery. The name of that file should contain the name of the folder it is describing.

It is recommended to store such a description file inside the folder it is describing rather than along with it in the parent directory, to make it easier to find if the items in the parent directory are not sorted alphabetically, since a file managers might separate files and folders regardless of which sorting method (alphabetical, by size, by name, by last modified) is chosen, which would make the description file not appear next to the folder it is describing.

== Observations and tips ==

=== Spare directories ===
When space on a device or partition is exhausted, no new directories that could be helpful for organizing files can be created, such as for moving files from a highly ''populated'' directory (i.e. with many files, such as a download folder) on a mobile phone in order to skip having to open the populated folder directly through [[#Media Transfer Protocol|MTP (Media Transfer Protocol)]], which notoriously handles long file listings poorly.

Prepare for such a situation by creating a reserve of spare empty directories inside one dedicated directory. The spare directories can be moved out of the reserve, and be renamed as necessary, even without space left, which allows organizing files on the go to be able to move them elsewhere immediately when arriving home.

=== File move behaviour ===
When the aim is to bring files to an archive, moving files rather than copying and deleting afterwards has the convenience benefit of acting like a check list of files, instead of creating duplicates that would later have to be sorted out without accidentally deleting non-copied files, as well as imminently clearing (freeing) space on the source device.

Moving instead of copying and then deleting files also defeats the psychological barrier that may come from the deletion step, as it feels like a destructive action even though it leads to the same result as moving between storage devices. Another barrier is the uncertainty of having inadvertently selected any file not copied.

When moving files, some file managers may delete files individually after transfer, while others only delete selected files from source only after the last file has finished transfering.

Windows Explorer uses the former method for mass storage devices, but the latter with Media Transfer Protocol. The Linux file manager ''Nemo'' always uses the former method.

With the latter method, any interruption would cancel the file transfer without having freed up any space on the source device.

=== Escape auto-closure ===
Some file managers such as Windows Explorer close themselves when detecting the removal (unmounting and/or physical unplugging) of a storage device, while others jump to the starting directory. Some file managers might do nothing.

The first case may be perceived as annoying, as it forces users to re-open the file manager and navigate all the way back to the previous directory.<ref>[https://www.sevenforums.com/general-discussion/148620-how-stop-windows-explorer-closing-removable-disks.html ''How to stop Windows Explorer closing for removable Disks!'' – SevenForums]</ref>

This can be prevented by opening a different device in the file manager before unplugging. After plugging the device back in, the previously opened directory can be navigated back to immediately through the navigation history, using the on-screen {{button|←}} button or <kbd>Alt</kbd>+<kbd>←</kbd> on the keyboard.

=== Using the command line ===
Command-line file operations can be logged for later reference by using the "verbose" switch <code>--verbose</code> or shorthand <code>-v</code> on the <code>cp</code> and <code>mv</code> commands in Linux and redirecting the output into a text file by appending <code>>>path/to/textfile.txt</code>. On Windows, file operations are outputted by default, meaning no "verbose" switch is necessary. On Linux, using <code>| tee -a path/to/textfile.txt</code> allows visibly printing out command line output in real-time (as usual) while logging into a text file simultaneously.

Autocompletion using the <kbd>↹Tab</kbd> key is widely supported on both Microsoft Windows and Linux/Unix-based operating systems and facilitates navigation by selecting file and folder names.<ref>[https://www.computerhope.com/tips/tip176.htm Use tab to autocomplete commands in the command line (ComputerHope, 2020-12-31)]</ref>

=== Shortcuts ===
Don't hesitate to use bookmarks to frequently accessed directories in your file manager and file picker dialogue (also known as "Open" and "Save as"). If they become unnecessary, they can be removed at any time.

For command-line use, familiarize yourself with environment variables, as they allow quicker navigation throughout your directories. They include <code>~</code> on Linux and <code>%user profile</code> (user home directory, equivalent to <code>C:\Users\''Username''\</code>) and <code>%temp%</code> (<code>C:\Users\''Username''\AppData\Local\temp\</code>).

Environment variables are also typically recognized by file pickers.

Additional shortcuts for command-line use can be specified as variables in the script that runs when starting the terminal, which in Linux is typically located at <code>~/.bashrc</code>. For example, <code>$dl</code> can be set to refer to the download folder by appending <code>dl=~/Downloads</code> to that file.

=== Spaces in file names ===
Keep in mind that space characters in file and path names can become a nuisance when entering paths, creating variables, selecting text, and navigating and auto-completing in a command prompt/terminal using the <kbd>↹Tab</kbd> key.

=== Cleanup ===
"Cleanup" may refer to moving files scattered around the storage and desktop in folders to improve overview, or deleting files like duplicates to clear free space. In the former case, refer to [[/Directory structures]] and consider moving files you are unsure where to move into a [[#Dumping_ground|dumping ground]]. In the latter case, as with [[Backup#lostBackup|lost backups]], first weigh effort against price, meaning here first consider whether the time and effort spent searching files to clean up really outweighs the storage space price.

It is not worth trying to search and eliminate duplicate files for the sake of it if no significant space is cleared. The time and effort necessary to do it might outweigh any benefit from the saved space. However, duplicates may be deleted for organizational purposes, such as duplicate music tracks in the same folder.

Note that cleaning does not necessarily speed up the computer noticeably, except if the partition was nearly full, which should be avoided anyway, as described in [[#Avoid_exhausting_the_operating_system's_partition|§ Avoid exhausting the operating system's partition]]. Rather, closing tasks that stress the CPU and take much RAM is effective in speeding up the system. Clearing caches is mostly unnecessary and even disadvantageous, as they provide faster recall of information.

For files vulnerable to damage like PNG and TIFF images, and [[Backup#Compressed_archives|compressed archives]], duplicates are beneficial for files not backed up yet.

== Experiences by participants ==
This section goes deeper into the idea about that "the average computer and mobile phone user still struggles to keep track of files in the long term despite of all the tools at their disposal". Here participants or visitors to this learning resource can add why they need a resource such as this to learn from and/or interact with it.
:''Moved to [[Talk:{{PAGENAME}}]].''

== References ==
<references />

== See also ==
* [[File naming]]
* [[File systems]]
* [[Backup]]
* [[Data recovery]]

== External links ==
* Comics:
** [https://xkcd.com/949/ File Transfer – xkcd #949, 2011] ([https://www.explainxkcd.com/wiki/index.php/949:_File_Transfer explainer])
** [https://xkcd.com/1360/ Old Files – xkcd #1360, 2014] ([https://www.explainxkcd.com/wiki/index.php/1360:_Old_Files explainer])
** [https://xkcd.com/1817/ Incognito Mode – xkcd #1817, 2017] ([https://www.explainxkcd.com/wiki/index.php/1817:_Incognito_Mode explainer])

[[Category:Information technology]]
[[Category:Computer_science]]

Backup

2024-10-22T12:34:29Z

Elominius: /* Classes of data */ clarification

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Classes of data ==
Consider classifying your data by their degree of importance. Data with a higher class should be backed up more frequently, and/or more redundantly.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts and projects you worked a long time on, photographs and home video recordings such as family-related and honeymoon, stuff you want to show to your future grandchildren, important emails, paid software license keys that are still redeemable, and helpful web resources that have been or are likely to be taken down.

'''Second-class data''' might be a full backup of your operating system that is periodically overwritten with a new backup anyway, photographs and video recordings that are not critical, and downloaded web pages or online videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection.

'''Third-class data''' are computer programs that can easily be downloaded again, temporary files such as [[:w:Archive_file|archive files]] (zip, 7z, tar) which were only created for sending the files contained within, and [[:w:Cache (computing)|cached data]]. Third-class data are also physical popular books and movies that can easily be re-purchased again anytime, or are stored on optical discs (CD, DVD, Blu-ray) which have a high resistance against failure.

Collections of data such as a digital music collection might be considered first- or second-class even though the individual files are replaceable, because a collection might have taken lots of time and effort to create.

First-class data should be backed up at least twice, meaning after which the data exists three times.

If feasible, consider creating an off-site backup of any data you consider first-class. This can be a friend's house or a cloud storage service. This protects the data from natural disasters.

Second-class data should be backed up at least once, though a backup is not urgent if the data is stored on new media that is highly unlikely to fail anytime soon, or have fewer points of failure, such as optical discs. Second-class data on storage media expected to fail soon should be moved onto new storage media.

Third-class data requires no backups since losing that data is not critical. Third-class data may also be only stored on a cloud storage service without a local copy.

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Backup/Causes of data loss

2024-10-22T12:27:40Z

Elominius: /* Failed writes */ loose connector

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== Failed writes ==
Data can get lost from unsuccessful writes to storage media. This can be caused by a full operating system partition, by unplugging external storage such as USB sticks without ejecting them safely, or by a loose connector that could unexpectedly disconnect an external storage device from the computer.

To improve performance, many operating systems use a write buffer. This means newly added files appear completely written to programs so they can be accessed faster, even if they are not completely written.

On Windows, a safe ejection can be performed from a tray icon at the right end of the task bar. On Linux, a <code>sync</code> command forces finishing writes, and the <code>umount</code> and <code>eject</code> commands followed by the device name or mountpoint can be used to prepare the computer for a disconnection of the device. The device name and mountpoint can be obtained using the <code>lsblk</code> (list block devices) command.

Some software might not be able to handle a no-space-left condition and corrupt or blank a file it writes to. In addition to loss of work, this can lead to reset preferences since the configuration file might get corrupted or blanked.

In rare occasions, an operating system might be rendered unbootable if its partition is completely full.<ref>[https://www.youtube.com/watch?v=UlOta2qYcm4 Das Apple Drama!! [VLOG 4]] - Kelly MissesVlog (2015-01-01) - YouTube (8 minutes)</ref> Then, the only solution is to boot from an external operating system installed on portable storage media such as an external SSD, then manually move files off the operating system partition to a different partition or device that has free space.

== Malicious software ==
Malicious software, also known as [[:wikipedia:malware|malware]], might damage your data while executing its payload. While early malware usually caused damage to entertain their creators, more modern malware known as ramsomware encrypts files and then demands a payment for decrypting the files.

If your data has been encrypted by ramsomware and you have no backup, the chance of getting it varies depending on the program. For more recent versions of ramsomware, there may be no chance of getting it back without paying the ramsom. Even then, you are not guaranteed to regain access to your files given that an anonymous cybercriminal gets to decide whether your files will be decrypted.<ref>{{cite web |url=https://old.reddit.com/r/techsupport/comments/loce4w/my_company_was_hacked_and_years_worth_of_files/ |title=My company was hacked and years worth of files have been encrypted. |date=2021-02-20 |access-date=2024-07-20 |website=Tech Support - Reddit |quote=After spending a month hiring companies to try and get it back they payed the ransom of a staggering 1500 dollars and got it all back.}} - the company had not renewed its backup for seven years, since 2014. </ref><ref>[https://www.nomoreransom.org/ The No More Ransom Project]</ref>

Consider storing very important data on [[:wikipedia:Write once read many|write-once media]] such as recordable optical discs (CD-R, DVD±R, BD-R), given that data on those media is inviolable by malware.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup

2024-10-22T12:21:24Z

Elominius: /* Classes of data */ rephrased

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Classes of data ==
Consider classifying your data by their degree of importance. Data with a higher class should be backed up more frequently, and/or more redundantly.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts and projects you worked a long time on, photographs and home video recordings such as family-related and honeymoon, stuff you want to show to your future grandchildren, important emails, paid software license keys, and helpful web resources that have been or are likely to be taken down.

'''Second-class data''' might be a full backup of your operating system that is periodically overwritten with a new backup anyway, photographs and video recordingss that are not critical, and downloaded web pages or online videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection.

'''Third-class data''' are computer programs that can easily be downloaded again, temporary files such as [[:w:Archive_file|archive files]] (zip, 7z, tar) which were only created for sending files contained within, or [[:w:Cache (computing)|cached data]]. Third-class data are also physical popular books and movies that can be re-purchased again anytime, or are stored on optical discs (CD, DVD, Blu-ray) which have a high resistance against failure.

Collections of data such as a digital music collection might be considered first- or second-class even though the individual files are replaceable, because a collection might have taken lots of time and effort to create.

First-class data should be backed up at least twice, meaning after which the data exists three times.

If feasible, consider creating an off-site backup of any data you consider first-class. This can be a friend's house or a cloud storage service. This protects the data from natural disasters.

Second-class data should be backed up at least once, though a backup is not urgent if the data is stored on new media that is highly unlikely to fail anytime soon, or have fewer points of failure, such as optical discs. Second-class data on storage media expected to fail soon should be moved onto new storage media.

Third-class data requires no backups since losing that data is not critical. Third-class data may also be only stored on a cloud storage service without a local copy.

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Backup

2024-10-22T12:20:28Z

Elominius: /* Classes of data */ more examples

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Classes of data ==
Consider classifying your data by their degree of importance. Data with a higher class should be backed up more frequently, and/or more redundantly.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts and projects you worked a long time on, photographs and home video recordings such as family-related and honeymoon, stuff you want to show to your future grandchildren, important emails, paid software license keys, and helpful web resources that have been or are likely to be taken down.

'''Second-class data''' might be a full backup of your operating system that is periodically overwritten with a new backup anyway, photographs and video recordingss that are not critical, and downloaded web pages or online videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection.

'''Third-class data''' are computer programs that can easily be downloaded again, temporary files such as [[:w:Archive_file|archive files]] (zip, 7z, tar) which were only created for sending files contained within, or [[:w:Cache (computing)|cached data]]. Third-class data are also physical popular books and movies that can be re-purchased again anytime, or are stored on optical discs (CD, DVD, Blu-ray) which have a high resistance against failure.

Collections of data that took lots of time to create, such as a digital music collection, might be considered first or second class even though the individual files are replaceable.

First-class data should be backed up at least twice, meaning after which the data exists three times.

If feasible, consider creating an off-site backup of any data you consider first-class. This can be a friend's house or a cloud storage service. This protects the data from natural disasters.

Second-class data should be backed up at least once, though a backup is not urgent if the data is stored on new media that is highly unlikely to fail anytime soon, or have fewer points of failure, such as optical discs. Second-class data on storage media expected to fail soon should be moved onto new storage media.

Third-class data requires no backups since losing that data is not critical. Third-class data may also be only stored on a cloud storage service without a local copy.

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Backup

2024-10-22T12:06:17Z

Elominius: /* Classes of data */ more examples

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Classes of data ==
Consider classifying your data by their degree of importance. Data with a higher class should be backed up more frequently, and/or more redundantly.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts you worked a long time on, photographs and home video recordings such as family-related and honeymoon, important emails, and helpful web resources that have been or are likely to be taken down.

'''Second-class data''' might be a full backup of your operating system that is periodically overwritten with a new backup anyway, photographs and videos that are not critical, and downloaded web pages or videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection.

'''Third-class data''' are computer programs that can easily be downloaded again, temporary files such as [[:w:Archive_file|archive files]] (zip, 7z, tar) which were only created for sending files contained within, or [[:w:Cache (computing)|cached data]].

First-class data should be backed up at least twice, meaning after which the data exists three times.

If feasible, consider creating an off-site backup of any data you consider first-class. This can be a friend's house or a cloud storage service. This protects the data from natural disasters.

Second-class data should be backed up at least once, though a backup is not urgent if the data is stored on new media that is highly unlikely to fail anytime soon, or have fewer points of failure, such as optical discs. Second-class data on storage media expected to fail soon should be moved onto new storage media.

Third-class data requires no backups since losing that data is not critical. Third-class data may also be only stored on a cloud storage service without a local copy.

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Backup

2024-10-22T11:55:31Z

Elominius: more suitable place; added content

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Classes of data ==
Consider classifying your data by their degree of importance. Data with a higher class should be backed up more frequently, and/or more redundantly.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts you worked a long time on, photographs and home videos such as family-related and honeymoon, important emails, and helpful web resources that have been or are likely to be taken down. '''Second-class data''' might be photographss and videos that are not critical, and downloaded web pages or videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection. '''Third-class data''' are computer programs that can easily be downloaded again, temporary files, or [[:w:Cache (computing)|cached data]].

First-class data should be backed up at least twice, meaning after which the data exists three times.

If feasible, consider creating an off-site backup of any data you consider first-class. This can be a friend's house or a cloud storage service. This protects the data from natural disasters.

Second-class data should be backed up at least once, though a backup is not urgent if the data is stored on new media that is highly unlikely to fail anytime soon, or have fewer points of failure, such as optical discs. Second-class data on storage media expected to fail soon should be moved onto new storage media.

Third-class data requires no backups since losing that data is not critical.

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Backup

2024-10-22T11:43:51Z

Elominius: /* Practices */ + classes of data

In information technology, a backup, or data backup, or the process of backing up, refers to the copying into an archive file of computer data that is already in secondary storage—so that it may be used to restore the original after a data loss event. The verb form is "back up" (a phrasal verb), whereas the noun and adjective form is "backup".<ref>[[Wikipedia: Backup]]</ref>

Backups primarily serve to restore the previous state after the loss, inadvertent deletion or corruption of data, and secondarily to recover data from an earlier time, based on a user-defined data retention policy.[3] Though backups represent a simple form of disaster recovery and should be part of any disaster recovery plan, backups by themselves should not be considered a complete disaster recovery plan (DRP). One reason for this is that not all backup systems are able to reconstitute a computer system or other complex configuration such as a computer cluster, active directory server, or database server by simply restoring data from a backup.<ref>[[Wikipedia: Backup]]</ref>

Not only is a backup worth it to allow for restoration in case of data loss, but it will also allow you to rest assured knowing that your files are safe, similarly to having to worry less about injury when wearing a helmet while riding a bycicle. Data loss might cause an uncertainty of which memorable data is lost, if its date and/or the date range of the lost data is unknown.
[[:w:Murphy's Law|Murphy's Law]] states that "if something can go wrong, it probably will". As such, any data one does not wish to lose should be stored on more than one device.

== Causes of data loss ==
Data may be lost as a result of:

* Device failure
* Software bugs and poorly designed user interfaces
* Software lock-in
* Human error
* Overreliance on online services

For a more detailed explanation, read the sub page [[Backup/Causes of data loss]].

== Types of data storage ==
=== Magnetic and optical ===
Various types of data storage have benefits and disadvantages. As such, optical discs have the most reliability for archival storage, and hard disks are the most suitable for routine short-term backups.

Optical and tape storage are not vulnerable to data loss from mechanical failure, due to their modularity, as their controller is external rather than tied to the component that holds the data. In case of motor or loading mechanism failure of an optical drive, inserting a dull needle into the pinhole can force opening the tray to retrieve the media. Slot-load drives may require opening should the loading mechanism fail.

; Short-term
For routine short-term backups, especially disk image backups, external mechanical hard drives are the most suitable, since such backups involve much sequential writing, which wears down hard disks far less than flash storage. Hard drives just alter magnetic fields when writing data, and the head moves very little when writing sequentially, where as flash storage has to erase and reprogram memory cells, which wears it down. In addition, hard disks cost less than flash storage.

Linear tape storage could also be used for routine short-term backups, however, it is expensive and not user-friendly, has a high learning curve, and thus a high barrier of entry.

; Long-term
[[File:QPxTool DVD error rate graph.png|thumb|If supported by the optical drive model, software is able to measure the rate of minor errors on optical discs. The errors depicted in this graph are normal and well within a correctable range and have not corrupted any data.]]
[[File:Water running on CD-RW - label side.jpg|thumb|Optical discs are water-resistant and therefore have a good chance of surviving a flood disaster. ]]
Optical media does not have the vulnerabilities of other storage media such as head crashes and controller failures, making it ideal for the sporadic low-maintenance long-term archival of moderately sized chunks of data such as video recordings from an important event, at capacities available as of 2023, meaning 25 GB per single-layer Blu-ray disc (least cost) and up to 128 GB per disc (multi-layer).

Higher-capacity optical formats such as the Sony Archival Disc exist, however, they are proprietary and vendor-exclusive, causing [[:w:vendor lock-in|vendor lock-in]].

Since optical discs are water-resistant and non-magnetic, they are the likeliest to survive a flood disaster or solar storm.

In addition, optical media can, if supported by the drive model, be scanned for impending integrity errors using software such as ''QpxTool'' and ''Nero DiscSpeed'', before any data loss occurs. A higher rate of still correctible errors suggests sooner data corruption, and/or media of lower quality.

Hard drive failure is mechanical and sudden, whereas optical media deterioration ("disc rot") happens slowly over time. Optical media is best stored in a cold and dry environment, whereas hard disk drives are less sensitive to heat and humidity while not in operation.<ref>https://smallbusiness.chron.com/temperature-should-laptop-hard-drive-run-81401.html</ref><ref>https://www.clir.org/pubs/reports/pub121/sec5/</ref> However, humidity should be avoided ''during'' hard drives' operation, as it increases the likelihood of failure.<ref>https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/</ref> ard drive motors' lubrication slowly loses effectiveness as well.<ref>http://www1.coe.neu.edu/~smuftu/docs/2011/ME5656_Term_Project%20Air%20lubrication%20in%20HDDs.PDF</ref> Hard drives' magnetic fields fade slowly over time, which could lead to logical data errors before the hard drive technically fails. For prevention, data would have to be rewritten once every few years. Proneness to such increases with drives' information density. The magnetic signal on magnetic tapes also fades over time, though manufacturers proclaim a "shelf life" of several decades for such.<ref>[https://searchdatabackup.techtarget.com/tip/How-to-estimate-the-lifespan-of-LTO-tapes How to estimate the lifespan of LTO tapes] – "30 years" ([http://www.fujifilmusa.com/shared/bin/LTO_Data_Tape_Seminar_2012.pdf source])</ref>

On hard disk drives, a slight possibility of manufacturing error leading to rapid failure exists.<ref>[https://www.kitguru.net/components/hard-drives/anton-shilov/hdd-expert-reliability-of-hdds-depends-on-manufacturing-process/ HDD Expert: Reliability of HDDs depends on manufacturing process – Anton Shilov, September 30, 2014, KitGuru]</ref> If a hard drive is intended to be used for long-term storage, it is recommended to only operate it infrequently once written to, to minimize mechanical wear.

Humidity in an enclosed storage location can be reduced using silica gel baglets.

Optical media is the most likely to survive environmental disasters such as a flood, ionizing radiation from nuclear disaster, or the marginal possibility of an electromagnetic/solar radiation storm,<ref>[https://www.washingtonexaminer.com/washington-secrets/military-warns-emp-attack-could-wipe-out-america-democracy-world-order ''Military warns EMP attack could wipe out America'']</ref><ref>[https://www.youtube.com/watch?v=oHHSSJDJ4oo ''Could Solar Storms Destroy Civilization? Solar Flares & Coronal Mass Ejections'' – Kurzgesagt - in a nutshell] (Video, 09m43s)</ref><ref>[https://www.gotquestions.org/EMP-attack.html ''Could an EMP attack be a part of the end times?'']</ref> as it is water-resistant and has no internal electrical components vulnerable to [[:w:single effect events|single effect events]].

There exists optical media that is dedicated to archival, using silver, gold, or rock-like layers rather than organic dye, making it less sensitive to environmental conditions.<ref>[https://www.verbatim.com/prod/optical-media/dvd/archival-grade-gold-dvd-r/ultralife/ "Verbatim Gold" (silver and gold)] and [https://www.m-disc.com/technology.html "M-Disc" (rock-like)]</ref>

=== Flash storage ===
Flash memory (solid state drives, USB sticks, memorybcards), while physically the most durable and usually fast, tend to be expensive per capacity, and might not be able to retain full data integrity for a long time (i.e. years), as the transistors which hold data in the form of electrical charge lose that charge over time.<ref name=ni12>{{cite web |date=2022-06-01 |title=Understanding Life Expectancy of Flash Storage |url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html |access-date=2022-10-18 |website=www.ni.com |language=en}}</ref> This is also referred to as "bit fading".

Flash memory can be used for supplementary backups and short-term storage. However, the price per capacity is higher on flash storage than magnetic and optical storage. The main use of flash storage is for portable electronics and operating systems due to their physical sturdiness and immediate random access.

The retention duration on flash storage tends to be shorter with increasing data density, and deteriorates throughout weardown caused by repeated program/erase cycles. Life expectancy (program/erase cycles) also deteriorates with data density. However, flash memory does not rely on moving mechanical parts that can fail like hard disk drives, hence the name "solid state drive". While USB sticks and memory cards are by definition also "solid state drives", the term is established to refer to larger-sized units for clarity. The lack of mechanical parts means that the flash memory does not wear down while idle unlike the spinning hard drive motor.

While powered on and idle, the flash storage's control firmware usually refreshes the information stored inside the sectors routinely.<ref>{{Cite web|date=2020-07-23|title=Understanding Life Expectancy of Flash Storage|url=https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html|access-date=2020-12-19|website=www.ni.com|language=en}}</ref>

Loss of data integrity is indicated by downspiking transfer rates caused by the flash memory controller attempting to correct errors.

Imminent failure of flash storage can be caused by power surges and voltage spikes. In comparison, hard disk drive controllers can also fail by such, but data stored on the metal disks, meaning a costly recovery may be possible, and optical discs can simply be retrieved as described earlier.

As flash storage is typically not intended for archival, and a possibility of component failure exists, anything stored on such that one does not wish to lose should be mirrored to at least one other location. For example, a large hard disk drive can be purposed to store corresponding disk images of flash media that are refreshed once in a while. This is especially recommended when storing a collection of frequently needed files or even an operating system on a flash drive, where scraping the files together or installing an operating system took time and effort, as a disk image allows sequentially reinstating the data onto the same or a different flash drive in case of error.

==== Memory cards ====
Mobile phone and tablet PC users may back up files short-term onto the removable memory card as an insurance against technical defect which denies access to its non-removable internal storage.

This can be useful for photography and filming during a trip where cloud storage would be impractical due to possibly limited transfer rates and data plans unable to handle high-resolution imagery, and the protrusion of a flash drive connected through ''USB On-The-Go'' would compromise necessary ergonomy.

== Online services ==
{{anchor|Cloud_storage}}
Online services like cloud storage are just an '''extension''' and not a '''substitute''' to local data storage. Since cloud storage is a remote service rather than a personally owned product, the end user lacks technical control. Services might have varying retention spans and inactivity policies,<ref>For example, [https://help.dropbox.com/accounts-billing/settings-sign-in/email-about-inactive-account Dropbox retains unused accounts for up to one year].</ref> and technical difficulties are not predictable to end users. Access requires internet connection, and transfer rates are limited by such. The slight possibility of erroneous account termination by a service provider exists as well.<ref>"Many people, incl. friends of mine, rely on ‘ cloud’ for file storage, use ‘ free’ messengers, listen to music on ‘ services’, and expect companies to provide stable and consistent service. This is what you get in the end: [GitHub account temporarily inaccessible]. I have always treated public services as a handout that could be taken away any moment, without any explanation, and their availability as a long-drawn moment between downtimes, data corruption or content deletion due to DMCA or other rules or laws, and favor self-hosted […]" — Twitter user ''@ValdikSS'', [https://twitter.com/ValdikSS/status/1116066219281592320 April 10, 2019 (multi-tweet)]</ref><ref>[https://karl-voit.at/cloud You Can't Control Your Data in the Cloud – Karl Voit]</ref> However, Cloud storage can act as a supplementary and short-term off-site backup, such as during vacation.

The same principles apply to other online services such as email and social media platforms. Make sure services you utilize are equipped with [[:w:Data portability|proper export functionality]].<ref>[https://agb-server.web.de/webdeagb?target=_blank Web.de mail terms of service with six-month retention policy]: "We reserve the right to purge user data of accounts not logged into for six months, without notice. Additionally, after a year of inactivity, we reserve the right to release your email address for new registration." Original text: "WEB.DE ist berechtigt, die im Account des Nutzers gespeicherten Nachrichten und sonstige Dateien nach einem Zeitraum von sechs Monaten der Inaktivität (kein Login über Webbrowser, Mail-App oder E-Mail-Programm) ohne Rückfrage zu löschen. Nach einem Zeitraum von einem Jahr der Inaktivität ist WEB.DE darüber hinaus berechtigt, die vom Nutzer bei WEB.DE registrierten E-Mail-Adressen freizugeben und anderen Nutzern zur Verfügung zu stellen."</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Video Creators TV</ref><ref>[https://www.factmag.com/2015/10/21/janet-jackson-will-have-your-instagram-permanently-deleted-if-you-post-pictures-of-her/ ''Janet Jackson will have your Instagram permanently deleted if you post tour videos'' (FactMag, October 21, 2015)]</ref>

Cloud storage may be less vulnerable to human error due to typically being managed by trained professionals, and takes the burden of maintenance away from the user. This is especially helpful to people with little [[file management]] knowledge.<ref>{{cite web |url=https://daringfireball.net/2014/09/security_tradeoffs |title=Security Trade-Offs |author=John Gruber |date=2014-09-03 |access-date=2023-01-21 }}</ref>

For people who wish to create an off-site backup but have no separate location other than their residence, cloud storage is an option. However, as described in the section below, there are privacy risks.

=== Privacy on cloud storage services ===
If you use a cloud storage service, you need to expect that the people operating the service reserve the ability to access anything you upload to their computers.
It does not necessarily mean they will, but that they technically are able to.

Cloud storage operators tend to be reluctant to store any data on their computers that they can not view themselves. Even if they claim to encrypt your data, they might reserve a master decryption key.

If you are lucky, no human employee at the cloud service provider stumbles upon your files. Popular services have millions of users with a sheer amount of data that can not be overviewed by human staff. However, it is not guaranteed never to occur. Some artificial intelligence might mark your files as "suspicious", for example due to suspected copyright violations, putting them in a queue for human review.<ref>Text files containing simple numbers such as "1" have been detected as copyright violations by Google Drive: {{cite web |title=Googles Algorithmen stufen Ziffern als Copyright-Verletzung ein |url=https://www.heise.de/news/Googles-Algorithmen-stufen-Ziffern-als-Copyright-Verletzung-ein-6338468.html |date=2022-01-26 |access-date=2023-01-21 |language=de |author=Daniel AJ Sokolov }}</ref>

== Environment ==
All data storage devices have a longer shelf life if stored in a cold and dry place such as a basement. For optical discs, the recommended temperature is at 15°C and 20°C (59°F and 68°F). Additionally, write-once optical media uses a light-sensitive dye, therefore is best stored in a dark location.<ref>[https://www.minimallstorage.com/blog/how-to-pack-and-store-dvds-and-disks-long-term/ ''How to Pack and Store DVDs and Discs Long-Term'' – Mini Mall Storage]</ref><ref>[https://www.clir.org/pubs/reports/pub121/sec5/ –
5. Conditions That Affect CDs and DVDs – Council on Library and Information Resources]</ref>

If there is no spare space for optical discs such as a separate spindle, the cover sheet that is typically included at the top of a disc spindle, usually for labeling, can instead be used in the time being to separate empty and written-to discs.

Hard drives are less sensitive to heat, but more sensitive to humidity, and last longer if stored and operated in dry air.<ref>{{cite web |url=https://www.zdnet.com/article/heat-doesnt-kill-hard-drives-heres-what-does/ |title= Heat doesn't kill hard drives. Here's what does |website=ZDNet |author=Robin Harris |date=2016-03-08 |access-date=2022-11-02 }}</ref><ref>{{cite web |url=https://www.backblaze.com/blog/hard-drive-temperature-does-it-matter/ |title=Hard Drive Temperature—Does It Matter? |website=BackBlaze |date=2014-05-12 |author=Brian Beach |access-date=2022-11-02 }}</ref>

Flash storage is not subject to mechanical wear down, but can retain data for a longer time at lower temperatures. Should it lose data due to bit fading as described in [[#Flash_storage|§ Flash storage]], it is not physically damaged and can still be re-used for new data. As already mentioned, long-term archival is not the design purpose of flash storage. The primary uses for flash storage are in portable devices and for hosting operating systems.<ref name=ni12 />

== Practices ==
=== Preparation ===
In a risky environment where there is an increased likelihood of data loss, such as a vaction or trip with the possibility of losing equipment, a higher backup frequency such as daily is recommended, which can be done at the base (hotel, holiday apartment, etc.) onto a portable hard disk drive or solid state drive.

For dedicated cameras and camcorders, memory cards can be cycled through.

=== Classes of data ===
You might want to classify your data by their level of importance.

'''First-class data''' is data that you never want to lose. '''Second-class data''' is data you prefer not to lose but losing them is not critical, or data that is relevant in the short term but you are unlikely to need in a decade. '''Third-class data''' are files that can easily be recreated or replaced or have no long-term value to you.

'''First-class data''' might be writings and scripts you worked a long time on, home videos such as family and honeymoon, important emails, and helpful web resources that have been or are likely to be taken down. '''Second-class data''' might be downloaded web pages or videos that are unlikely to be taken down, which you might have saved to read or watch them in places with no internet connection. '''Third-class data''' are computer programs that can easily be downloaded again, temporary files, or [[:w:Cache (computing)|cached data]].

Data with a higher class should be backed up more frequently, and/or more redundantly. Consider creating an off-site backup of data you consider first-class. Third-class data requires no backups since losing that data is not critical.

=== File system structure image ===
For users who momentarily lack space storage for backups, an image of merely the file system structure, which contains information about file names, paths, fragments and time attributes, can significantly facilitate later data recovery in case of damage. Without this information, any damage affecting the file system header could lead to files being ''orphaned'' and only detectable by forensic software through file headers and footers. Fragmented files would need to be [[File_puzzling|puzzled together]].

The file system structure (or ''header'') is usually stored in the first 50 to 200 Megabytes, which can be captured using disk imaging software within seconds.

While such a backup does not contain file contents (except possibly those located at the earliest logical block addresses (''LBAs'') shortly after the file system header itself), it is a fallback solution which is better than nothing.

=== Verifying backups ===
Depending on the importance of data, backups should be verified once every few years. If at least three copies exist, one verification per decade should suffice. If one of the backups is no longer properly readable, a new one on fresh storage media can be created.

The easiest way to verify backups and archives is to test whether a sample of randomly picked files is readable.

For archive file formats such as ZIP and 7z and GZip, archive managers such as ''7-Zip'' on Windows and ''File Roller'' on Linux have a feature for verifying data integrity and listing corrupt files. For entire devices such as hard drives and optical discs, tools such as the <code>badblocks</code> command on Linux and the freemium [[:w:shareware|shareware]] "IsoBuster" on Windows can be used to scan for unreadable sectors. On IsoBuster, sector scanning is part of the free features. These tools list unreadable sectors as numbers.

Another way to verify data integrity is by measuring the sequential reading speed. On Windows, the freeware tool "HDD scan" is able to generate a line graph of the reading speed. Ideally, the line is smooth. On flash storage, it ideally is straight and flat with minimal downspikes at most, and on spinning media such as hard drives and optical discs, the speed slightly decreases or increases due to the changing circumference per rotation where speeds are higher at the outer edge. Significant downspikes in speed are a hint to damage since the data storage controller has to spend effort correcting errors. For optical media, error rate scanning is available as described in [[#Magnetic_and_optical|§ Magnetic and optical]].

Unused flash storage would have to be verified at least yearly to prevent losing data, since the process of reading causes the flash memory controller to refresh the data.<ref name=ni12 /> However, as stated earlier ([[#Flash_storage|§ Flash storage]]), flash memory is not ideal for archival to begin with due to bit fading and higher price per capacity.

=== Cloning ===
Cloning is the practice of copying each bit from one data storage device sequentially to another. On Windows, this task can be performed with third-party software and on Linux with the <code>dd</code> command-line tool. Special caution should be taken not to select the wrong target device.

If your computer has a modular, replaceable data storage device, it can be replaced with a cloned data storage device in case of failure, allowing for resuming work with only minimal time loss.<ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com |access-date=2023-02-08 |quote=In case of an internal HD crash and failure, there is absolutely nothing quicker to getting back to 100% operation than having a HD clone handy to either boot from, or within 20 mins. installing and removing the bad HD. Nothing to install software-wise, and a speedy immediate return to your computer use and productivity. }}</ref>

=== Other tips ===
* {{Visible anchor|lostBackup|text='''Lost backups:'''}} If you are unable to physically locate a data storage device which contains a backup, act as if it were lost and consider just creating another backup from the source. The time and effort spared by foregoing to search for the lost device might be worth it. In case it is found again, one has one more redundant backup. If the redundant backup is not necessary, one of the storage devices can be re-purposed.
** The same applies to individual files you are not sure where you have already backed them up to. Back them up again, and to prevent losing track of files, create [[File management/Directory structures|logical structures of directories]]. In effect, the accessible copy of a file "seeds" that file to a new backup so you have at least two accessible copies.
** If you are certain that the data is backed up on a different device, you can store a redundant copy in a folder named "duplicates" or "possible duplicates" for the time being.
* {{Visible anchor|distinctBackup|text='''Distinct backup media:'''}} Backup media distinct from its source media prevents failure at a similar time from a common cause such as a manufacturing error.
* {{Visible anchor|lastbackup|text='''Tracking manual backups:'''}} Manual backups of individual directories can be tracked using a text file such as <code>lastbackup.txt</code> or <code>backupinfo.txt</code>, containing the time of the last backup and optionally additional information such as the name of the backup device.
* {{Visible anchor|reputableBrands|text='''Reputable brands:'''}} Data storage not from reputable and trustworthy brands or pre-owned should not be relied upon in long term, only for temporary storage, testing purposes, and similar.
* {{Visible anchor|proprietaryFormats|text='''Proprietary formats:'''}} Software that stores backups in proprietary formats such as that of Acronis (<code>.tib</code>) should be used with caution due to the possibility of [[:w:vendor lock-in|vendor lock-in]]. Should said vendor cease operations, it could become even more difficult to access archives in proprietary formats in the long term.
* {{Visible anchor|encryption|text='''Encryption:'''}} In the long term, encrypted backups and archives pose a risk of forgetting or losing the access credential (e.g. password). When using encryption, weigh the likelihood of physically losing the data storage against aforementioned risk, and factor in the sensitivity of data. For example, public web downloads usually don't need to be encrypted.

== Compressed archives ==
=== Compression ratio ===
Compressed archives may be used where efficient, such as for text documents and code, where strong compression formats such as 7-Zip (LZMA) and XZip could reduce size by a factor of 100 or more. Uncompressed bitmap images can achieve compression ratios of around 10; more or less depending on content.

Binary data such as multimedia (picture, video, audio) that has already been internally compressed can not be effectively shrunk by applying additional compression to it, as most redundancy has already been defeated by internal compression.

Which level of compression one should use depends on the contents. For example, for creating an archive file containing pre-compressed data such as PNG files, compression can be deactivated entirely for the highest performance, since data does not have to be encoded and later decoded during extraction. For text-based data, a high compression level is highly effective. For archiving disk image files and mixed data with varying levels of compressibility, weak compression is recommended for a minimal impact on performance while still significantly compressing blank parts ("[[:w:Solid_compression#Explanation|air]]") of the image and text-based files. Performance matters on disk image files due to their typically large size.

=== Magnification of damage ===
[[File:Aicp homescreen.bitflip glitch art.rerender.png|thumb|Damage on a PNG image from flipping a single bit]]
It should however be considered that the slightest damage to a compressed archive file could magnify enormously, possibly rendering the rest of an inside file or the entire archive useless. The scope of the damage depends on compression method, where [[:w:Solid compression|solid compression]] causes the latter, as it deduplicates information across contained files to maximize compression effectiveness. As such, it is recommended to store compressed archives on no less than two devices.

If a separate device is unavailable, storing a duplicate of the archive file on the same device for redundancy could still allow repairing errors in uncommon locations by merging the intact parts using a byte editor (or "hex editor"), though errors are difficult to locate if the storage device controller's firmware returns damaged data to the computer without reporting it as such. Some flash storage devices may return sectors (512 bytes each) with damaged data as null bytes. In that case, a multiple of 512 sequential null bytes between binary data suggests a logical error in the file. On optical discs, a sector typically has 2048 bytes.

To get a visualized idea of how fragile compressed archives can be to damage, try opening a PNG or TIFF image inside a byte editor (or ''hex editor''), and edit only few bytes somewhere near the middle, and then try the same on an uncompressed bitmap (BMP) for comparison. The PNG and TIFF images are to be completely demolished and glitched out from the damaged point, while effects on the uncompressed bitmap are only pinhole-sized.

This experiment might not be as effective on a JPEG image as on PNG and TIFF, as its compression algorithm is more robust against damage. It may cause some gitching and hue alterations on ordinary JPEG, and ''digital stains'' on progressive JPEG, but nothing that demolishes the entire image beyond repair. For reference, see the [[:Commons: category: Bit-blending]] experiment.

Compression formats that have a weaker ratio, but in return require far less computing efforts, might use block-based storage, where information is stored in compressed blocks of a fixed size. Data blocks after a damaged one might be recoverable. A popular block-based compression format is Gzip; see [[:w:Gzip#Damage_recovery|Gzip § Damage recovery]].

=== Efficiency ===
Human-readable text and code is highly compressible, and archive formats' internal file systems typically handle a high number of small files more efficiently than data storage file systems.

On FAT32/16/12 and exFAT for example, any non-empty file reserves at least one entire cluster (space allocation unit), which may be preformatted to around 16 to 256 KB, depending on total storage size. Too many small files cause space being wasted through ''cluster overhead'', whereas archive formats handle many small files efficiently, even with compression deactivated.

However, digital photographs and video are internally compressed to a degree where additional compression through an archive format such as ''Zip'', ''RAR'', and ''7z'' would not shrink the size effectively while significantly slowing down extraction speeds rather than allowing for direct playback, and making it vulnerable to damage.

Note that a ''[[:w:program stream|program stream]]'' video (most mobile video) with an end-of-file ''[[:w:moov atom|moov atom]]'' depends on completeness or else be unplayable. This is not the case for ''[[:w:transport stream|transport stream]]'' video, frequently used by dedicated camcorders such as those of Panasonic and Sony ([[:w:AVCHD|AVCHD]]).

== Preventing data lock-in ==
Wherever you store your data, verify in advance that proper export functionality exists.

For example, as of 2022, the email service "ProtonMail" lacks support for email protocols like POP3 or IMAP. The only way to back up messages is individually downloading them, which is impractical, since individually downloading one thousand emails could take several hours of repetitive clicking.

On the Android mobile operating system, applications might store user data inside the <code>/data/</code> folder. This includes web browsing session, history, and saved pages depending on the browser. This makes it only accessible to the application itself. In other for any other application to access it, [[:w:Rooting (Android)|root access]] must be unlocked.<ref>[https://beepb00p.xyz/exports.html Building data liberation infrastructure – Beepb00p.xyz]</ref>

== Format obsolescence ==
There is a slight likelihood that formats used today will become unsupported in future.

A past example is the HD-DVD, an optical disc format which [[:w:High_definition_optical_disc_format_war|did not succeed in the market]]. Unlike DVD plus and minus formats, HD-DVD could not successfully co-exist with its competing format, the Blu-ray disc, and production of HD-DVD media and hardware was halted. As a result, people might have recorded data on HD-DVD discs, but no longer own a drive that reads them. A Blu-ray drive could technically read HD-DVDs, since they are technically very similar and use the same wave length, but optical drive manufacturers see no need in implementing HD-DVD support.

== Other areas ==
=== Web browsing ===
==== Sessions ====
Develop the habit of exporting your browsing session into a text file regularly, which can be done through a browser extension. The automatic session restoration might fail, and the session database might have a proprietary format only readable or decodeable through difficult-to-use tools rather than a simple text editor.

Some browser extensions allow exporting both page title and URL, facilitating later searching. Some extensions have an option to limit the export to only tabs of the current window, which could be of use if tabs in other windows have not been changed.

Some browser extensions have an internal session manager, while others export the session into a file in the download folder. Some might have both. Some extensions for this purpose have been discarded as a result of Mozilla Firefox, one of the most popular web browsers, deprecating support for extensions in a legacy format with the transition to version 57 "Quantum", though functionally similar surrogates are presumed to be released.

==== Web forms ====
Web form data may get lost as a result of browser crashes caused by RAM exhaustion, operating system crashes or power outages without uninterrupted power supply (external unit or laptop battery), or a failed form submission.<ref>Data from failed form submissions may be recoverable through a "core dump" generated from the browser's process, for which you may refer to [https://superuser.com/questions/236390/how-do-i-recover-a-form-in-firefox-without-installing-a-plugin/236400#236400 this guide by Joey Adams].</ref>

Browser extensions such as ''Textarea Cache'' prevent loss of form data by backing it up automatically.<ref>[https://addons.mozilla.org/en-US/firefox/addon/textarea-cache/ Textarea Cache for Firefox]</ref>

Sites for which this is not wanted can be added to an exclusion list in the extension's settings.

Consider drafting any text that is presumed to take minutes or longer to write into an offline text file, from which it can be copied and pasted into a web form.

The same also applies to other areas such as to programs' internal text areas like the WinRAR archive comment field<ref>https://documentation.help/WinRAR/HELPArcComment.htm</ref>, since it may be discarded in case of an error or cancellation. Regarding WinRAR, using proprietary formats is not recommended anyway due to potential incompatibility throughout systems.

==== Browsing history ====
Web browsers might automatically delete history under certain conditions such as space storage exhaustion on the partition that stores the user data folder (or "profile folder"), or history from earlier than a time threshold such as three months. An update might unexpectedly change the retention duration, discarding history falling outside if shortened.<ref name="obsidian">{{cite web|last1=Benson|first1=Ryan|title=Archived History files removed from Chrome v37|url=http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|archive-url=https://web.archive.org/web/20141010125418/http://www.obsidianforensics.com/blog/archived-history-files-removed-from-chrome-v37/|url-status=dead|archive-date=2014-10-10|publisher=Obsidian Forensics|ref=obsidian}}</ref>

If you wish to retain history beyond browsers' retention span, consider routinely creating copies of the history database. Firefox's live database file is named <code>places.sqlite</code> and located in the profile folder. For Google Chrome/Chromium, extensions exist that allow exporting it into the download folder.<ref>[https://chrome.google.com/webstore/detail/export-historybookmarks-t/dcoegfodcnjofhjfbhegcgjgapeichlf Export History/Bookmarks to JSON/CSV*/XLS*] extension from the Google Chrome web store</ref>

Note that on mobile phones, [[File_management#On-device_management|restrictions by operating system]] could deny file-level access to browsers' user data. Then, the only way to export browsing data is using a mobile browser which supports third-party extensions, such as the Chromium fork "Kiwi browser", which is compatible with many desktop Google Chrome extensions.<ref>[https://kiwibrowser.com/features/ Features - Kiwi Browser]</ref>

=== Paper ===
Reliable long-term storage of relatively low amounts of data onto paper is possible using software such as ''[http://ollydbg.de/Paperbak/ PaperBak]'' ([https://github.com/Rupan/paperbak open-source]), which can encode digital data into printable machine-readable matrix bar code.

Paper books can not be duplicated as simply as digital computer files, but have supreme longevity and reliability due to immunity from technical failure. Abstractly, paper is a form of optical media.

Backing up paper books themselves may be desirable to protect against short-term losses such as forgetting to take them after use at a location such as school, university, or work place. Pages can be individually digitized in a high quality using optical flatbed scanners, though the process is slow and demands patience. It is faster and more conveniently done using a digital camera, preferably monuted to a tripod pointing down for a fixed position of the pages within the images, and in good lighting. Pairs of pages can be photographed individually, though filming while quickly skimming through the pages is the fastest method. The necessary resolution for clear readability depends on size and clarity of font, but at least 1080p is recommended, where each still frame of the video acts as a 2.1-Megapixel photograph, only with slight quality losses from video compression.

== Commercial Backup products ==
Different commercial products provide Backup and recovery capabilities such as [[Commvault]], [[Veritas NetBackup]] and [[Veritas Backup Exec]], [[Veeam Backup & Replication]], [[Arcserve]].

== Activities ==
# Read about [[w:NDMP]] (''Network Data Management Protocol'')
# Learn about [[RPO]] and [[RTO]]

== See Also ==
* [[Data recovery]]
* [[Database Fundamentals/Backup and Restore]]
* [[Oracle Database Administration/Backup and Recovery]]
* [[Linux server administration/Backup and Restore]]
* [[Disk arrays|Disk Arrays]]
* [[Wikipedia: Backup]]

== References ==
{{reflist}}

[[Category:Backup| ]]
[[Category:Information technology]]

Bash programming/Function Usage

2024-10-02T22:48:25Z

Elominius: /* Examples */ filecount, findfile.

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.
* Count the files in a directory:
** <code>filecount() { ls "$@" |wc -l; }</code>
* Search for files in a directory:
** <code>findfile() { find "$2" |grep -i "$1"; }</code>
=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will replace the video files, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:v copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"</code>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Backup/Causes of data loss

2024-09-30T15:00:25Z

Elominius: /* Failed writes */ +

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== Failed writes ==
Data can get lost from unsuccessful writes to storage media. This can be caused by a full operating system partition or by unplugging external storage such as USB sticks without ejecting them safely.

To improve performance, many operating systems use a write buffer. This means newly added files appear completely written to programs so they can be accessed faster, even if they are not completely written.

On Windows, a safe ejection can be performed from a tray icon at the right end of the task bar. On Linux, a <code>sync</code> command forces finishing writes, and the <code>umount</code> and <code>eject</code> commands followed by the device name or mountpoint can be used to prepare the computer for a disconnection of the device. The device name and mountpoint can be obtained using the <code>lsblk</code> (list block devices) command.

Some software might not be able to handle a no-space-left condition and corrupt or blank a file it writes to. In addition to loss of work, this can lead to reset preferences since the configuration file might get corrupted or blanked.

In rare occasions, an operating system might be rendered unbootable if its partition is completely full.<ref>[https://www.youtube.com/watch?v=UlOta2qYcm4 Das Apple Drama!! [VLOG 4]] - Kelly MissesVlog (2015-01-01) - YouTube (8 minutes)</ref> Then, the only solution is to boot from an external operating system installed on portable storage media such as an external SSD, then manually move files off the operating system partition to a different partition or device that has free space.

== Malicious software ==
Malicious software, also known as [[:wikipedia:malware|malware]], might damage your data while executing its payload. While early malware usually caused damage to entertain their creators, more modern malware known as ramsomware encrypts files and then demands a payment for decrypting the files.

If your data has been encrypted by ramsomware and you have no backup, the chance of getting it varies depending on the program. For more recent versions of ramsomware, there may be no chance of getting it back without paying the ramsom. Even then, you are not guaranteed to regain access to your files given that an anonymous cybercriminal gets to decide whether your files will be decrypted.<ref>{{cite web |url=https://old.reddit.com/r/techsupport/comments/loce4w/my_company_was_hacked_and_years_worth_of_files/ |title=My company was hacked and years worth of files have been encrypted. |date=2021-02-20 |access-date=2024-07-20 |website=Tech Support - Reddit |quote=After spending a month hiring companies to try and get it back they payed the ransom of a staggering 1500 dollars and got it all back.}} - the company had not renewed its backup for seven years, since 2014. </ref><ref>[https://www.nomoreransom.org/ The No More Ransom Project]</ref>

Consider storing very important data on [[:wikipedia:Write once read many|write-once media]] such as recordable optical discs (CD-R, DVD±R, BD-R), given that data on those media is inviolable by malware.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Wikiversity:What-goes-where 2024

2024-09-21T09:22:59Z

Elominius: /* The question of permanent residence for Draft:Foobar */ red box always appears

{{shortcut|WV:WGW2024}}
-----
The premise of this page is that policy decisions on page deletions are coupled into other questions: When do we move pages into draft-space, user-space, or place them into subpages of namespace? It's all one big conundrum.
-----
{{center|''At the moment there I am the only contributor to this page. <br>You may edit or add sections as you wish. <br>We can work out the organizational details later'' --[[User:Guy vandegrift|Guy vandegrift]]}}
-----
==The question of permanent residence for [[Draft:Foobar]]==
We need to resolve the question of whether resources can remain indefinitely in draft-space. Recent discussions have been held in three places:
:#Wikiversity:Colloquium {{permalink|2616624#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely}}
:#Wikiversity talk:Drafts {{permalink|2615120#policy_and_page_change_suggestion}}
:#Wikiversity talk:Deletions {{Permalink|2612981#Proposed_modifications}}
From these three pages, [[User:Guy vandegrift|I]] count the following "votes" on the motion to revert the 6-month rule and allow articles in draft-space to remain indefinitely:
*'''4 vote in Favor''' of permanent residence for articles in draft-space: [[User:Guy vandegrift]], [[User:Michael Ten]], [[User:Jtneill]], [[User:Addemf]]
*'''1 is Skeptical''': [[User:Dan Polansky]]
* '''1 Opposes''' the motion: [[User:koavf]]

Is a 4-1-1 vote a consensus? Was this even a vote, since I inferred these "votes" from remarks the editors made. If and when we get a proper vote, should a bureaucrat close it? For what it's worth, the vote is 2-0-1 in favor if you count members of the ''[[Wikiversity:Support staff]].'' Also, the [[User:Guy vandegrift|''person'']] doing this counting is very bad at arithmetic, so I can't even guarantee that the vote tally is correct.
:"Is a 4-1-1 vote a consensus?" IMO until there is more participation, yes. i am biased though. if others join the project who care enough to voice an opinion later... the guideline/policy can always be changed back so that good faith Creative Commons contributions start being regularly deleted then, yes? limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 19:24, 31 March 2024 (UTC)

Drafts deleted too soon might never get the chance to flourish by being taken over by a new editor. It would be more productive if they could immediately resume on existing work rather than starting over or asking for the restoration of a draft. As far as I know, Wikipedia drafts are deleted after half a year because their draft space is only sparsely monitored, but Wikiversity is not close to the size of Wikipedia, so that is not an issue for us.

I think inactive drafts should simply be deposited into the author's user space so they can keep working on it should they come back, or another editor finds it through the light red notice saying the page has been moved.

If someone is concerned about drafts being confused with main space resources, perhaps a box at the top, such as [[template:draft]], would prevent drafts for being mistaken as main space resources, but I think it is very unlikely to happen anyway. Someone who is able to find a page in user space will surely be able to distinguish user space from main space. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:04, 14 April 2024 (UTC) - 22:15, 14 April 2024 (UTC)

:I agree that inactive drafts belong in the author's userspace. The problem is with articles that do not belong in mainspace that were written by multiple authors. This is the problem that [[Draft:Archive]] is designed to solve.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 23:44, 14 April 2024 (UTC)
::In case of multiple authors, a possibility would be the page creator's user space, or the user space of the editor who contributed the most text.
::If a page was moved away without leaving a redirect, a red box shows where it was moved so it can be found by anyone who might want to work on it. Users can also opt to adopt drafts by moving them from an inactive author's user space into their own. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 00:42, 20 July 2024 (UTC)
:::It never occurred to me that the red box could solve the problem. But, I visited [[Test page]] after logging out, the box does not appear. I suspect that only certain people (Curators, ect.) can see such moves when they click the link. Do you know? [[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 02:11, 21 July 2024 (UTC)
::::(Apologies for late response, I didn't log in for a while.) The red box actually appears to me both while logged in and logged out. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 09:22, 21 September 2024 (UTC)

==Terminology and Flowchart==
[[File:Wikiversity page location flowchart.svg|thumb|'''Purging''' draft-archived pages dewikifies incoming links, category statements, and templates. Unpurged versions are viewable via the history (see [[Draft:Archive/2024/Adobe Illustrator]].) <br><br>
'''Soft deletion''' is deletion with the understanding that the page will be undeleted upon request. A collection of soft deletions can be found at
[[User:Guy vandegrift/Soft deletions]]]]
*'''top-namespace''': Google searches top-mainspace articles like [[Foobar]]. Such pages can also be found by clicking "Random" on Wikiversity.
*'''sub-namespace''': [[Foo Studies/Foobar]] is also searched by Google, which can bypass the parent page and reach subpages, causing the child to get more visits than the parent [https://pageviews.wmcloud.org/?project=en.wikiversity.org&platform=all-access&agent=user&redirects=0&start=2022-03&end=2024-02&pages=Physics_equations|Physics_equations/Current_and_current_density (example.)]
*'''user-space''' [[User:John Doe/Foobar]] is a destination for a move of [[Foobar]] out of mainspace whenever [[User:John Doe]] is the sole author. I don't think user-space is searched by Google (can anybody verify?)
*'''draft-space''' was initially intended to be a construction site for [[Foobar]], but recently we have been moving [[Foobar]] to [[Draft:Foobar]] in order to get the page out of namespace if it has multiple authors (e.g., [[User:Jane Doe]] in the flowchart shown.)
*'''draft-archive''' [[Draft:Archive/2024/Foobar]] is an [[wikt:add-hoc|add-hoc]] space created by requests to get [[Foobar]] out of draft-space. One advantage of draft-archive is that all such pages are adjacent to each other. This makes it easy to '''purge''' archived pages of entities such as [[Wikiversity:Categories|category membership]]. '''A purged and archived page can be seen at ''[[ Draft:Archive/2024/Adobe Illustrator]]'''''.
*'''Hard and soft deletions:''' Pages that violate obvious rules (e.g., copyright) are easily identified and can be deleted immediately in what might be called a ''hard'' deletion. ''Soft'' deletion is identical to the ''hard'', except that the user understands that the page can be undeleted upon request. One way to achieve this is to maintain a list of pages that have been soft deleted. We also need to tell users where they can find this list (see this '''collection of files have been "soft" deleted from a userspace at ''[[User:Guy vandegrift/Soft deletions]]'''''
===Too many choices?===
It is possible that this flowchart diagram gives us too many choices that it will difficult for our community to reach a consensus and take action. For that reason, we should consider narrowing our option from those presented the flowchart.

Having said that we might have too many choices, there is another option that we can include without cluttering up the rules: Apparently it is possible to store a document on Google in a way that permits discovery via a Google search:
*[https://www.google.com/search?q=Socialism+site%3Adocs.google.com&rlz=1C1CHBF_enUS1007US1007&sxsrf=ALiCzsbf_f6YqprH9EPmJ9uYKX0U5vJH_Q%3A1672527697188&ei=Ub-wY_CMC_qeqtsP6bK2sA0&ved=0ahUKEwiw-ZGa-6T8AhV6j2oFHWmZDdYQ4dUDCBE&uact=5&oq=Socialism+site%3Adocs.google.com&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWMoUYO0baABwAHgAgAFPiAGeBJIBATmYAQCgAQHAAQE&sclient=gws-wiz-serp#ip=1 Socialism Google search with keywords site:docs.google.com]
*[https://docs.google.com/document/preview?hgd=1&id=1fzTVcIzElIMqioRVgudahXcSD32v79bz4AKHfVpOQkg Socialism essay stored by Google]

I do not know if this service is provided for free; I vaguely recall setting my Google docs to private when I created my own (free) account. But if this is possible, we certainly should advise users of this option when we tell them they can't have something in mainspace.

==Personal subpages (with visual editing)==
The "Wikiversity:" namespace does not provide visual editing. Here you can create a personal subspace with visual editing. It can also be used as a personal edit space that you can link to from this discussion. It is completely editable, and there is no reason to sign everything you like. And it has a talk page so others can comment on your ideas.

It is not necessary to name the personal userpage with your username (i.e., [[User:Foobar]] uses ''Foobar''.) Also, ''Guest'' is just that: You are free to play with it, or if you don't have a personal subpage, you can use ''Guest'' to draft complicated text before entering into even draft a witty reply before entering it into [[Wikiversity:What-goes-where 2024]]

<inputbox>
type=create
prefix=Draft:Wikiversity:What-goes-where 2024/
editintro=Book Reviews/Preload
preload=Wikiversity:What-goes-where 2024/PersonalSubpagePreload
default=
bgcolor=#f8fcff
buttonlabel=Create subpage
width=80
</inputbox>

;List of personal subpages

{{colbegin}}
{{Special:Prefixindex/Draft:Wikiversity:What-goes-where 2024/|hideredirects=0|stripprefix=1}}
{{colend}}

[[Category:Wikiversity deletion]]

Backup/Causes of data loss

2024-07-20T01:00:41Z

Elominius: website parameter

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== Malicious software ==
Malicious software, also known as [[:wikipedia:malware|malware]], might damage your data while executing its payload. While early malware usually caused damage to entertain their creators, more modern malware known as ramsomware encrypts files and then demands a payment for decrypting the files.

If your data has been encrypted by ramsomware and you have no backup, the chance of getting it varies depending on the program. For more recent versions of ramsomware, there may be no chance of getting it back without paying the ramsom. Even then, you are not guaranteed to regain access to your files given that an anonymous cybercriminal gets to decide whether your files will be decrypted.<ref>{{cite web |url=https://old.reddit.com/r/techsupport/comments/loce4w/my_company_was_hacked_and_years_worth_of_files/ |title=My company was hacked and years worth of files have been encrypted. |date=2021-02-20 |access-date=2024-07-20 |website=Tech Support - Reddit |quote=After spending a month hiring companies to try and get it back they payed the ransom of a staggering 1500 dollars and got it all back.}} - the company had not renewed its backup for seven years, since 2014. </ref><ref>[https://www.nomoreransom.org/ The No More Ransom Project]</ref>

Consider storing very important data on [[:wikipedia:Write once read many|write-once media]] such as recordable optical discs (CD-R, DVD±R, BD-R), given that data on those media is inviolable by malware.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup/Causes of data loss

2024-07-20T00:58:29Z

Elominius: /* Malicious software */ missing comma

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== Malicious software ==
Malicious software, also known as [[:wikipedia:malware|malware]], might damage your data while executing its payload. While early malware usually caused damage to entertain their creators, more modern malware known as ramsomware encrypts files and then demands a payment for decrypting the files.

If your data has been encrypted by ramsomware and you have no backup, the chance of getting it varies depending on the program. For more recent versions of ramsomware, there may be no chance of getting it back without paying the ramsom. Even then, you are not guaranteed to regain access to your files given that an anonymous cybercriminal gets to decide whether your files will be decrypted.<ref>{{cite web |url=https://old.reddit.com/r/techsupport/comments/loce4w/my_company_was_hacked_and_years_worth_of_files/ |title=My company was hacked and years worth of files have been encrypted. |date=2021-02-20 |access-date=2024-07-20 |quote=After spending a month hiring companies to try and get it back they payed the ransom of a staggering 1500 dollars and got it all back.}} - the company had not renewed its backup for seven years, since 2014. </ref><ref>[https://www.nomoreransom.org/ The No More Ransom Project]</ref>

Consider storing very important data on [[:wikipedia:Write once read many|write-once media]] such as recordable optical discs (CD-R, DVD±R, BD-R), given that data on those media is inviolable by malware.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup/Causes of data loss

2024-07-20T00:58:04Z

Elominius: Malware was already mentioned but deserves a section of its own.

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== Malicious software ==
Malicious software, also known as [[:wikipedia:malware|malware]] might damage your data while executing its payload. While early malware usually caused damage to entertain their creators, more modern malware known as ramsomware encrypts files and then demands a payment for decrypting the files.

If your data has been encrypted by ramsomware and you have no backup, the chance of getting it varies depending on the program. For more recent versions of ramsomware, there may be no chance of getting it back without paying the ramsom. Even then, you are not guaranteed to regain access to your files given that an anonymous cybercriminal gets to decide whether your files will be decrypted.<ref>{{cite web |url=https://old.reddit.com/r/techsupport/comments/loce4w/my_company_was_hacked_and_years_worth_of_files/ |title=My company was hacked and years worth of files have been encrypted. |date=2021-02-20 |access-date=2024-07-20 |quote=After spending a month hiring companies to try and get it back they payed the ransom of a staggering 1500 dollars and got it all back.}} - the company had not renewed its backup for seven years, since 2014. </ref><ref>[https://www.nomoreransom.org/ The No More Ransom Project]</ref>

Consider storing very important data on [[:wikipedia:Write once read many|write-once media]] such as recordable optical discs (CD-R, DVD±R, BD-R), given that data on those media is inviolable by malware.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Wikiversity:What-goes-where 2024

2024-07-20T00:42:27Z

Elominius: /* The question of permanent residence for Draft:Foobar */ Reply

{{shortcut|WV:WGW2024}}
-----
The premise of this page is that policy decisions on page deletions are coupled into other questions: When do we move pages into draft-space, user-space, or place them into subpages of namespace? It's all one big conundrum.
-----
{{center|''At the moment there I am the only contributor to this page. <br>You may edit or add sections as you wish. <br>We can work out the organizational details later'' --[[User:Guy vandegrift|Guy vandegrift]]}}
-----
==The question of permanent residence for [[Draft:Foobar]]==
We need to resolve the question of whether resources can remain indefinitely in draft-space. Recent discussions have been held in three places:
:#Wikiversity:Colloquium {{permalink|2616624#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely}}
:#Wikiversity talk:Drafts {{permalink|2615120#policy_and_page_change_suggestion}}
:#Wikiversity talk:Deletions {{Permalink|2612981#Proposed_modifications}}
From these three pages, [[User:Guy vandegrift|I]] count the following "votes" on the motion to revert the 6-month rule and allow articles in draft-space to remain indefinitely:
*'''4 vote in Favor''' of permanent residence for articles in draft-space: [[User:Guy vandegrift]], [[User:Michael Ten]], [[User:Jtneill]], [[User:Addemf]]
*'''1 is Skeptical''': [[User:Dan Polansky]]
* '''1 Opposes''' the motion: [[User:koavf]]

Is a 4-1-1 vote a consensus? Was this even a vote, since I inferred these "votes" from remarks the editors made. If and when we get a proper vote, should a bureaucrat close it? For what it's worth, the vote is 2-0-1 in favor if you count members of the ''[[Wikiversity:Support staff]].'' Also, the [[User:Guy vandegrift|''person'']] doing this counting is very bad at arithmetic, so I can't even guarantee that the vote tally is correct.
:"Is a 4-1-1 vote a consensus?" IMO until there is more participation, yes. i am biased though. if others join the project who care enough to voice an opinion later... the guideline/policy can always be changed back so that good faith Creative Commons contributions start being regularly deleted then, yes? limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 19:24, 31 March 2024 (UTC)

Drafts deleted too soon might never get the chance to flourish by being taken over by a new editor. It would be more productive if they could immediately resume on existing work rather than starting over or asking for the restoration of a draft. As far as I know, Wikipedia drafts are deleted after half a year because their draft space is only sparsely monitored, but Wikiversity is not close to the size of Wikipedia, so that is not an issue for us.

I think inactive drafts should simply be deposited into the author's user space so they can keep working on it should they come back, or another editor finds it through the light red notice saying the page has been moved.

If someone is concerned about drafts being confused with main space resources, perhaps a box at the top, such as [[template:draft]], would prevent drafts for being mistaken as main space resources, but I think it is very unlikely to happen anyway. Someone who is able to find a page in user space will surely be able to distinguish user space from main space. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:04, 14 April 2024 (UTC) - 22:15, 14 April 2024 (UTC)

:I agree that inactive drafts belong in the author's userspace. The problem is with articles that do not belong in mainspace that were written by multiple authors. This is the problem that [[Draft:Archive]] is designed to solve.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 23:44, 14 April 2024 (UTC)
::In case of multiple authors, a possibility would be the page creator's user space, or the user space of the editor who contributed the most text.
::If a page was moved away without leaving a redirect, a red box shows where it was moved so it can be found by anyone who might want to work on it. Users can also opt to adopt drafts by moving them from an inactive author's user space into their own. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 00:42, 20 July 2024 (UTC)

==Terminology and Flowchart==
[[File:Wikiversity page location flowchart.svg|thumb|'''Purging''' draft-archived pages dewikifies incoming links, category statements, and templates. Unpurged versions are viewable via the history (see [[Draft:Archive/2024/Adobe Illustrator]].) <br><br>
'''Soft deletion''' is deletion with the understanding that the page will be undeleted upon request. A collection of soft deletions can be found at
[[User:Guy vandegrift/Soft deletions]]]]
*'''top-namespace''': Google searches top-mainspace articles like [[Foobar]]. Such pages can also be found by clicking "Random" on Wikiversity.
*'''sub-namespace''': [[Foo Studies/Foobar]] is also searched by Google, which can bypass the parent page and reach subpages, causing the child to get more visits than the parent [https://pageviews.wmcloud.org/?project=en.wikiversity.org&platform=all-access&agent=user&redirects=0&start=2022-03&end=2024-02&pages=Physics_equations|Physics_equations/Current_and_current_density (example.)]
*'''user-space''' [[User:John Doe/Foobar]] is a destination for a move of [[Foobar]] out of mainspace whenever [[User:John Doe]] is the sole author. I don't think user-space is searched by Google (can anybody verify?)
*'''draft-space''' was initially intended to be a construction site for [[Foobar]], but recently we have been moving [[Foobar]] to [[Draft:Foobar]] in order to get the page out of namespace if it has multiple authors (e.g., [[User:Jane Doe]] in the flowchart shown.)
*'''draft-archive''' [[Draft:Archive/2024/Foobar]] is an [[wikt:add-hoc|add-hoc]] space created by requests to get [[Foobar]] out of draft-space. One advantage of draft-archive is that all such pages are adjacent to each other. This makes it easy to '''purge''' archived pages of entities such as [[Wikiversity:Categories|category membership]]. '''A purged and archived page can be seen at ''[[ Draft:Archive/2024/Adobe Illustrator]]'''''.
*'''Hard and soft deletions:''' Pages that violate obvious rules (e.g., copyright) are easily identified and can be deleted immediately in what might be called a ''hard'' deletion. ''Soft'' deletion is identical to the ''hard'', except that the user understands that the page can be undeleted upon request. One way to achieve this is to maintain a list of pages that have been soft deleted. We also need to tell users where they can find this list (see this '''collection of files have been "soft" deleted from a userspace at ''[[User:Guy vandegrift/Soft deletions]]'''''
===Too many choices?===
It is possible that this flowchart diagram gives us too many choices that it will difficult for our community to reach a consensus and take action. For that reason, we should consider narrowing our option from those presented the flowchart.

Having said that we might have too many choices, there is another option that we can include without cluttering up the rules: Apparently it is possible to store a document on Google in a way that permits discovery via a Google search:
*[https://www.google.com/search?q=Socialism+site%3Adocs.google.com&rlz=1C1CHBF_enUS1007US1007&sxsrf=ALiCzsbf_f6YqprH9EPmJ9uYKX0U5vJH_Q%3A1672527697188&ei=Ub-wY_CMC_qeqtsP6bK2sA0&ved=0ahUKEwiw-ZGa-6T8AhV6j2oFHWmZDdYQ4dUDCBE&uact=5&oq=Socialism+site%3Adocs.google.com&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWMoUYO0baABwAHgAgAFPiAGeBJIBATmYAQCgAQHAAQE&sclient=gws-wiz-serp#ip=1 Socialism Google search with keywords site:docs.google.com]
*[https://docs.google.com/document/preview?hgd=1&id=1fzTVcIzElIMqioRVgudahXcSD32v79bz4AKHfVpOQkg Socialism essay stored by Google]

I do not know if this service is provided for free; I vaguely recall setting my Google docs to private when I created my own (free) account. But if this is possible, we certainly should advise users of this option when we tell them they can't have something in mainspace.

==Personal subpages (with visual editing)==
The "Wikiversity:" namespace does not provide visual editing. Here you can create a personal subspace with visual editing. It can also be used as a personal edit space that you can link to from this discussion. It is completely editable, and there is no reason to sign everything you like. And it has a talk page so others can comment on your ideas.

It is not necessary to name the personal userpage with your username (i.e., [[User:Foobar]] uses ''Foobar''.) Also, ''Guest'' is just that: You are free to play with it, or if you don't have a personal subpage, you can use ''Guest'' to draft complicated text before entering into even draft a witty reply before entering it into [[Wikiversity:What-goes-where 2024]]

<inputbox>
type=create
prefix=Draft:Wikiversity:What-goes-where 2024/
editintro=Book Reviews/Preload
preload=Wikiversity:What-goes-where 2024/PersonalSubpagePreload
default=
bgcolor=#f8fcff
buttonlabel=Create subpage
width=80
</inputbox>

;List of personal subpages

{{colbegin}}
{{Special:Prefixindex/Draft:Wikiversity:What-goes-where 2024/|hideredirects=0|stripprefix=1}}
{{colend}}

[[Category:Wikiversity deletion]]

Wikiversity:What-goes-where 2024

2024-04-14T22:15:25Z

Elominius: no confusion of drafts with main space resources

{{shortcut|WV:WGW2024}}
-----
The premise of this page is that policy decisions on page deletions are coupled into other questions: When do we move pages into draft-space, user-space, or place them into subpages of namespace? It's all one big conundrum.
-----
{{center|''At the moment there I am the only contributor to this page. <br>You may edit or add sections as you wish. <br>We can work out the organizational details later'' --[[User:Guy vandegrift|Guy vandegrift]]}}
-----
==The question of permanent residence for [[Draft:Foobar]]==
We need to resolve the question of whether resources can remain indefinitely in draft-space. Recent discussions have been held in three places:
:#Wikiversity:Colloquium {{permalink|2616624#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely}}
:#Wikiversity talk:Drafts {{permalink|2615120#policy_and_page_change_suggestion}}
:#Wikiversity talk:Deletions {{Permalink|2612981#Proposed_modifications}}
From these three pages, [[User:Guy vandegrift|I]] count the following "votes" on the motion to revert the 6-month rule and allow articles in draft-space to remain indefinitely:
*'''4 vote in Favor''' of permanent residence for articles in draft-space: [[User:Guy vandegrift]], [[User:Michael Ten]], [[User:Jtneill]], [[User:Addemf]]
*'''1 is Skeptical''': [[User:Dan Polansky]]
* '''1 Opposes''' the motion: [[User:koavf]]

Is a 4-1-1 vote a consensus? Was this even a vote, since I inferred these "votes" from remarks the editors made. If and when we get a proper vote, should a bureaucrat close it? For what it's worth, the vote is 2-0-1 in favor if you count members of the ''[[Wikiversity:Support staff]].'' Also, the [[User:Guy vandegrift|''person'']] doing this counting is very bad at arithmetic, so I can't even guarantee that the vote tally is correct.
:"Is a 4-1-1 vote a consensus?" IMO until there is more participation, yes. i am biased though. if others join the project who care enough to voice an opinion later... the guideline/policy can always be changed back so that good faith Creative Commons contributions start being regularly deleted then, yes? limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 19:24, 31 March 2024 (UTC)

Drafts deleted too soon might never get the chance to flourish by being taken over by a new editor. It would be more productive if they could immediately resume on existing work rather than starting over or asking for the restoration of a draft. As far as I know, Wikipedia drafts are deleted after half a year because their draft space is only sparsely monitored, but Wikiversity is not close to the size of Wikipedia, so that is not an issue for us.

I think inactive drafts should simply be deposited into the author's user space so they can keep working on it should they come back, or another editor finds it through the light red notice saying the page has been moved.

If someone is concerned about drafts being confused with main space resources, perhaps a box at the top, such as [[template:draft]], would prevent drafts for being mistaken as main space resources, but I think it is very unlikely to happen anyway. Someone who is able to find a page in user space will surely be able to distinguish user space from main space. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:04, 14 April 2024 (UTC) - 22:15, 14 April 2024 (UTC)

==Terminology and Flowchart==
[[File:Wikiversity page location flowchart.svg|thumb|'''Purging''' draft-archived pages dewikifies incoming links, category statements, and templates. Unpurged versions are viewable via the history (see [[Draft:Archive/2024/Adobe Illustrator]].) <br><br>
'''Soft deletion''' is deletion with the understanding that the page will be undeleted upon request. A collection of soft deletions can be found at
[[User:Guy vandegrift/Soft deletions]]]]
*'''top-namespace''': Google searches top-mainspace articles like [[Foobar]]. Such pages can also be found by clicking "Random" on Wikiversity.
*'''sub-namespace''': [[Foo Studies/Foobar]] is also searched by Google, which can bypass the parent page and reach subpages, causing the child to get more visits than the parent [https://pageviews.wmcloud.org/?project=en.wikiversity.org&platform=all-access&agent=user&redirects=0&start=2022-03&end=2024-02&pages=Physics_equations|Physics_equations/Current_and_current_density (example.)]
*'''user-space''' [[User:John Doe/Foobar]] is a destination for a move of [[Foobar]] out of mainspace whenever [[User:John Doe]] is the sole author. I don't think user-space is searched by Google (can anybody verify?)
*'''draft-space''' was initially intended to be a construction site for [[Foobar]], but recently we have been moving [[Foobar]] to [[Draft:Foobar]] in order to get the page out of namespace if it has multiple authors (e.g., [[User:Jane Doe]] in the flowchart shown.)
*'''draft-archive''' [[Draft:Archive/2024/Foobar]] is an [[wikt:add-hoc|add-hoc]] space created by requests to get [[Foobar]] out of draft-space. One advantage of draft-archive is that all such pages are adjacent to each other. This makes it easy to '''purge''' archived pages of entities such as [[Wikiversity:Categories|category membership]]. '''A purged and archived page can be seen at ''[[ Draft:Archive/2024/Adobe Illustrator]]'''''.
*'''Hard and soft deletions:''' Pages that violate obvious rules (e.g., copyright) are easily identified and can be deleted immediately in what might be called a ''hard'' deletion. ''Soft'' deletion is identical to the ''hard'', except that the user understands that the page can be undeleted upon request. One way to achieve this is to maintain a list of pages that have been soft deleted. We also need to tell users where they can find this list (see this '''collection of files have been "soft" deleted from a userspace at ''[[User:Guy vandegrift/Soft deletions]]'''''
===Too many choices?===
It is possible that this flowchart diagram gives us too many choices that it will difficult for our community to reach a consensus and take action. For that reason, we should consider narrowing our option from those presented the flowchart.

Having said that we might have too many choices, there is another option that we can include without cluttering up the rules: Apparently it is possible to store a document on Google in a way that permits discovery via a Google search:
*[https://www.google.com/search?q=Socialism+site%3Adocs.google.com&rlz=1C1CHBF_enUS1007US1007&sxsrf=ALiCzsbf_f6YqprH9EPmJ9uYKX0U5vJH_Q%3A1672527697188&ei=Ub-wY_CMC_qeqtsP6bK2sA0&ved=0ahUKEwiw-ZGa-6T8AhV6j2oFHWmZDdYQ4dUDCBE&uact=5&oq=Socialism+site%3Adocs.google.com&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWMoUYO0baABwAHgAgAFPiAGeBJIBATmYAQCgAQHAAQE&sclient=gws-wiz-serp#ip=1 Socialism Google search with keywords site:docs.google.com]
*[https://docs.google.com/document/preview?hgd=1&id=1fzTVcIzElIMqioRVgudahXcSD32v79bz4AKHfVpOQkg Socialism essay stored by Google]

I do not know if this service is provided for free; I vaguely recall setting my Google docs to private when I created my own (free) account. But if this is possible, we certainly should advise users of this option when we tell them they can't have something in mainspace.

==Personal subpages (with visual editing)==
The "Wikiversity:" namespace does not provide visual editing. Here you can create a personal subspace with visual editing. It can also be used as a personal edit space that you can link to from this discussion. It is completely editable, and there is no reason to sign everything you like. And it has a talk page so others can comment on your ideas.

It is not necessary to name the personal userpage with your username (i.e., [[User:Foobar]] uses ''Foobar''.) Also, ''Guest'' is just that: You are free to play with it, or if you don't have a personal subpage, you can use ''Guest'' to draft complicated text before entering into even draft a witty reply before entering it into [[Wikiversity:What-goes-where 2024]]

<inputbox>
type=create
prefix=Draft:Wikiversity:What-goes-where 2024/
editintro=Book Reviews/Preload
preload=Wikiversity:What-goes-where 2024/PersonalSubpagePreload
default=
bgcolor=#f8fcff
buttonlabel=Create subpage
width=80
</inputbox>

;List of personal subpages

{{colbegin}}
{{Special:Prefixindex/Draft:Wikiversity:What-goes-where 2024/|hideredirects=0|stripprefix=1}}
{{colend}}

[[Category:Wikiversity deletion]]

Wikiversity:What-goes-where 2024

2024-04-14T22:08:00Z

Elominius: more plausible

{{shortcut|WV:WGW2024}}
-----
The premise of this page is that policy decisions on page deletions are coupled into other questions: When do we move pages into draft-space, user-space, or place them into subpages of namespace? It's all one big conundrum.
-----
{{center|''At the moment there I am the only contributor to this page. <br>You may edit or add sections as you wish. <br>We can work out the organizational details later'' --[[User:Guy vandegrift|Guy vandegrift]]}}
-----
==The question of permanent residence for [[Draft:Foobar]]==
We need to resolve the question of whether resources can remain indefinitely in draft-space. Recent discussions have been held in three places:
:#Wikiversity:Colloquium {{permalink|2616624#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely}}
:#Wikiversity talk:Drafts {{permalink|2615120#policy_and_page_change_suggestion}}
:#Wikiversity talk:Deletions {{Permalink|2612981#Proposed_modifications}}
From these three pages, [[User:Guy vandegrift|I]] count the following "votes" on the motion to revert the 6-month rule and allow articles in draft-space to remain indefinitely:
*'''4 vote in Favor''' of permanent residence for articles in draft-space: [[User:Guy vandegrift]], [[User:Michael Ten]], [[User:Jtneill]], [[User:Addemf]]
*'''1 is Skeptical''': [[User:Dan Polansky]]
* '''1 Opposes''' the motion: [[User:koavf]]

Is a 4-1-1 vote a consensus? Was this even a vote, since I inferred these "votes" from remarks the editors made. If and when we get a proper vote, should a bureaucrat close it? For what it's worth, the vote is 2-0-1 in favor if you count members of the ''[[Wikiversity:Support staff]].'' Also, the [[User:Guy vandegrift|''person'']] doing this counting is very bad at arithmetic, so I can't even guarantee that the vote tally is correct.
:"Is a 4-1-1 vote a consensus?" IMO until there is more participation, yes. i am biased though. if others join the project who care enough to voice an opinion later... the guideline/policy can always be changed back so that good faith Creative Commons contributions start being regularly deleted then, yes? limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 19:24, 31 March 2024 (UTC)

Drafts deleted too soon might never get the chance to flourish by being taken over by a new editor. It would be more productive if they could immediately resume on existing work rather than starting over or asking for the restoration of a draft. As far as I know, Wikipedia drafts are deleted after half a year because their draft space is only sparsely monitored, but Wikiversity is not close to the size of Wikipedia, so that is not an issue for us.

I think inactive drafts should simply be deposited into the author's user space so they can keep working on it should they come back, or another editor finds it through the light red notice saying the page has been moved. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:04, 14 April 2024 (UTC) - last modified 22:07, 14 April 2024 (UTC)

==Terminology and Flowchart==
[[File:Wikiversity page location flowchart.svg|thumb|'''Purging''' draft-archived pages dewikifies incoming links, category statements, and templates. Unpurged versions are viewable via the history (see [[Draft:Archive/2024/Adobe Illustrator]].) <br><br>
'''Soft deletion''' is deletion with the understanding that the page will be undeleted upon request. A collection of soft deletions can be found at
[[User:Guy vandegrift/Soft deletions]]]]
*'''top-namespace''': Google searches top-mainspace articles like [[Foobar]]. Such pages can also be found by clicking "Random" on Wikiversity.
*'''sub-namespace''': [[Foo Studies/Foobar]] is also searched by Google, which can bypass the parent page and reach subpages, causing the child to get more visits than the parent [https://pageviews.wmcloud.org/?project=en.wikiversity.org&platform=all-access&agent=user&redirects=0&start=2022-03&end=2024-02&pages=Physics_equations|Physics_equations/Current_and_current_density (example.)]
*'''user-space''' [[User:John Doe/Foobar]] is a destination for a move of [[Foobar]] out of mainspace whenever [[User:John Doe]] is the sole author. I don't think user-space is searched by Google (can anybody verify?)
*'''draft-space''' was initially intended to be a construction site for [[Foobar]], but recently we have been moving [[Foobar]] to [[Draft:Foobar]] in order to get the page out of namespace if it has multiple authors (e.g., [[User:Jane Doe]] in the flowchart shown.)
*'''draft-archive''' [[Draft:Archive/2024/Foobar]] is an [[wikt:add-hoc|add-hoc]] space created by requests to get [[Foobar]] out of draft-space. One advantage of draft-archive is that all such pages are adjacent to each other. This makes it easy to '''purge''' archived pages of entities such as [[Wikiversity:Categories|category membership]]. '''A purged and archived page can be seen at ''[[ Draft:Archive/2024/Adobe Illustrator]]'''''.
*'''Hard and soft deletions:''' Pages that violate obvious rules (e.g., copyright) are easily identified and can be deleted immediately in what might be called a ''hard'' deletion. ''Soft'' deletion is identical to the ''hard'', except that the user understands that the page can be undeleted upon request. One way to achieve this is to maintain a list of pages that have been soft deleted. We also need to tell users where they can find this list (see this '''collection of files have been "soft" deleted from a userspace at ''[[User:Guy vandegrift/Soft deletions]]'''''
===Too many choices?===
It is possible that this flowchart diagram gives us too many choices that it will difficult for our community to reach a consensus and take action. For that reason, we should consider narrowing our option from those presented the flowchart.

Having said that we might have too many choices, there is another option that we can include without cluttering up the rules: Apparently it is possible to store a document on Google in a way that permits discovery via a Google search:
*[https://www.google.com/search?q=Socialism+site%3Adocs.google.com&rlz=1C1CHBF_enUS1007US1007&sxsrf=ALiCzsbf_f6YqprH9EPmJ9uYKX0U5vJH_Q%3A1672527697188&ei=Ub-wY_CMC_qeqtsP6bK2sA0&ved=0ahUKEwiw-ZGa-6T8AhV6j2oFHWmZDdYQ4dUDCBE&uact=5&oq=Socialism+site%3Adocs.google.com&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWMoUYO0baABwAHgAgAFPiAGeBJIBATmYAQCgAQHAAQE&sclient=gws-wiz-serp#ip=1 Socialism Google search with keywords site:docs.google.com]
*[https://docs.google.com/document/preview?hgd=1&id=1fzTVcIzElIMqioRVgudahXcSD32v79bz4AKHfVpOQkg Socialism essay stored by Google]

I do not know if this service is provided for free; I vaguely recall setting my Google docs to private when I created my own (free) account. But if this is possible, we certainly should advise users of this option when we tell them they can't have something in mainspace.

==Personal subpages (with visual editing)==
The "Wikiversity:" namespace does not provide visual editing. Here you can create a personal subspace with visual editing. It can also be used as a personal edit space that you can link to from this discussion. It is completely editable, and there is no reason to sign everything you like. And it has a talk page so others can comment on your ideas.

It is not necessary to name the personal userpage with your username (i.e., [[User:Foobar]] uses ''Foobar''.) Also, ''Guest'' is just that: You are free to play with it, or if you don't have a personal subpage, you can use ''Guest'' to draft complicated text before entering into even draft a witty reply before entering it into [[Wikiversity:What-goes-where 2024]]

<inputbox>
type=create
prefix=Draft:Wikiversity:What-goes-where 2024/
editintro=Book Reviews/Preload
preload=Wikiversity:What-goes-where 2024/PersonalSubpagePreload
default=
bgcolor=#f8fcff
buttonlabel=Create subpage
width=80
</inputbox>

;List of personal subpages

{{colbegin}}
{{Special:Prefixindex/Draft:Wikiversity:What-goes-where 2024/|hideredirects=0|stripprefix=1}}
{{colend}}

[[Category:Wikiversity deletion]]

Wikiversity:What-goes-where 2024

2024-04-14T22:04:32Z

Elominius: Adding my thoughts.

{{shortcut|WV:WGW2024}}
-----
The premise of this page is that policy decisions on page deletions are coupled into other questions: When do we move pages into draft-space, user-space, or place them into subpages of namespace? It's all one big conundrum.
-----
{{center|''At the moment there I am the only contributor to this page. <br>You may edit or add sections as you wish. <br>We can work out the organizational details later'' --[[User:Guy vandegrift|Guy vandegrift]]}}
-----
==The question of permanent residence for [[Draft:Foobar]]==
We need to resolve the question of whether resources can remain indefinitely in draft-space. Recent discussions have been held in three places:
:#Wikiversity:Colloquium {{permalink|2616624#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely}}
:#Wikiversity talk:Drafts {{permalink|2615120#policy_and_page_change_suggestion}}
:#Wikiversity talk:Deletions {{Permalink|2612981#Proposed_modifications}}
From these three pages, [[User:Guy vandegrift|I]] count the following "votes" on the motion to revert the 6-month rule and allow articles in draft-space to remain indefinitely:
*'''4 vote in Favor''' of permanent residence for articles in draft-space: [[User:Guy vandegrift]], [[User:Michael Ten]], [[User:Jtneill]], [[User:Addemf]]
*'''1 is Skeptical''': [[User:Dan Polansky]]
* '''1 Opposes''' the motion: [[User:koavf]]

Is a 4-1-1 vote a consensus? Was this even a vote, since I inferred these "votes" from remarks the editors made. If and when we get a proper vote, should a bureaucrat close it? For what it's worth, the vote is 2-0-1 in favor if you count members of the ''[[Wikiversity:Support staff]].'' Also, the [[User:Guy vandegrift|''person'']] doing this counting is very bad at arithmetic, so I can't even guarantee that the vote tally is correct.
:"Is a 4-1-1 vote a consensus?" IMO until there is more participation, yes. i am biased though. if others join the project who care enough to voice an opinion later... the guideline/policy can always be changed back so that good faith Creative Commons contributions start being regularly deleted then, yes? limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 19:24, 31 March 2024 (UTC)

Drafts deleted too soon might never get the chance to flourish by being taken over by a new editor. It would be more productive if they could immediately resume on existing work rather than starting over or asking for the restoration of a draft. As far as I know, Wikipedia drafts are deleted after half a year to reduce the workload of editors who maintain draft articles, but Wikiversity is not close to the size of Wikipedia, so that is not an issue for us.

I think inactive drafts should simply be deposited into the author's user space so they can keep working on it should they come back, or another editor finds it through the light red notice saying the page has been moved. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:04, 14 April 2024 (UTC)

==Terminology and Flowchart==
[[File:Wikiversity page location flowchart.svg|thumb|'''Purging''' draft-archived pages dewikifies incoming links, category statements, and templates. Unpurged versions are viewable via the history (see [[Draft:Archive/2024/Adobe Illustrator]].) <br><br>
'''Soft deletion''' is deletion with the understanding that the page will be undeleted upon request. A collection of soft deletions can be found at
[[User:Guy vandegrift/Soft deletions]]]]
*'''top-namespace''': Google searches top-mainspace articles like [[Foobar]]. Such pages can also be found by clicking "Random" on Wikiversity.
*'''sub-namespace''': [[Foo Studies/Foobar]] is also searched by Google, which can bypass the parent page and reach subpages, causing the child to get more visits than the parent [https://pageviews.wmcloud.org/?project=en.wikiversity.org&platform=all-access&agent=user&redirects=0&start=2022-03&end=2024-02&pages=Physics_equations|Physics_equations/Current_and_current_density (example.)]
*'''user-space''' [[User:John Doe/Foobar]] is a destination for a move of [[Foobar]] out of mainspace whenever [[User:John Doe]] is the sole author. I don't think user-space is searched by Google (can anybody verify?)
*'''draft-space''' was initially intended to be a construction site for [[Foobar]], but recently we have been moving [[Foobar]] to [[Draft:Foobar]] in order to get the page out of namespace if it has multiple authors (e.g., [[User:Jane Doe]] in the flowchart shown.)
*'''draft-archive''' [[Draft:Archive/2024/Foobar]] is an [[wikt:add-hoc|add-hoc]] space created by requests to get [[Foobar]] out of draft-space. One advantage of draft-archive is that all such pages are adjacent to each other. This makes it easy to '''purge''' archived pages of entities such as [[Wikiversity:Categories|category membership]]. '''A purged and archived page can be seen at ''[[ Draft:Archive/2024/Adobe Illustrator]]'''''.
*'''Hard and soft deletions:''' Pages that violate obvious rules (e.g., copyright) are easily identified and can be deleted immediately in what might be called a ''hard'' deletion. ''Soft'' deletion is identical to the ''hard'', except that the user understands that the page can be undeleted upon request. One way to achieve this is to maintain a list of pages that have been soft deleted. We also need to tell users where they can find this list (see this '''collection of files have been "soft" deleted from a userspace at ''[[User:Guy vandegrift/Soft deletions]]'''''
===Too many choices?===
It is possible that this flowchart diagram gives us too many choices that it will difficult for our community to reach a consensus and take action. For that reason, we should consider narrowing our option from those presented the flowchart.

Having said that we might have too many choices, there is another option that we can include without cluttering up the rules: Apparently it is possible to store a document on Google in a way that permits discovery via a Google search:
*[https://www.google.com/search?q=Socialism+site%3Adocs.google.com&rlz=1C1CHBF_enUS1007US1007&sxsrf=ALiCzsbf_f6YqprH9EPmJ9uYKX0U5vJH_Q%3A1672527697188&ei=Ub-wY_CMC_qeqtsP6bK2sA0&ved=0ahUKEwiw-ZGa-6T8AhV6j2oFHWmZDdYQ4dUDCBE&uact=5&oq=Socialism+site%3Adocs.google.com&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWMoUYO0baABwAHgAgAFPiAGeBJIBATmYAQCgAQHAAQE&sclient=gws-wiz-serp#ip=1 Socialism Google search with keywords site:docs.google.com]
*[https://docs.google.com/document/preview?hgd=1&id=1fzTVcIzElIMqioRVgudahXcSD32v79bz4AKHfVpOQkg Socialism essay stored by Google]

I do not know if this service is provided for free; I vaguely recall setting my Google docs to private when I created my own (free) account. But if this is possible, we certainly should advise users of this option when we tell them they can't have something in mainspace.

==Personal subpages (with visual editing)==
The "Wikiversity:" namespace does not provide visual editing. Here you can create a personal subspace with visual editing. It can also be used as a personal edit space that you can link to from this discussion. It is completely editable, and there is no reason to sign everything you like. And it has a talk page so others can comment on your ideas.

It is not necessary to name the personal userpage with your username (i.e., [[User:Foobar]] uses ''Foobar''.) Also, ''Guest'' is just that: You are free to play with it, or if you don't have a personal subpage, you can use ''Guest'' to draft complicated text before entering into even draft a witty reply before entering it into [[Wikiversity:What-goes-where 2024]]

<inputbox>
type=create
prefix=Draft:Wikiversity:What-goes-where 2024/
editintro=Book Reviews/Preload
preload=Wikiversity:What-goes-where 2024/PersonalSubpagePreload
default=
bgcolor=#f8fcff
buttonlabel=Create subpage
width=80
</inputbox>

;List of personal subpages

{{colbegin}}
{{Special:Prefixindex/Draft:Wikiversity:What-goes-where 2024/|hideredirects=0|stripprefix=1}}
{{colend}}

[[Category:Wikiversity deletion]]

User talk:Michael Ten

2024-04-11T21:05:09Z

Elominius: /* Changing support from draftspace to draft archive */ agreed

== Internet Protocol Analysis ==__NOTOC__

Thanks for stopping by and editing the intro to [[Internet_protocol_analysis | Internet Protocol Analysis]]. If you have any other recommendations, I'd appreciate hearing from you. The course goes live with approximately 100 real world students on January 14. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|talk]]) 23:39, 26 December 2012 (UTC)

== Earning money ==

What learning project is [[Earning money]] associated with? That title doesn't appear to be either an academic topic or have a corresponding Wikipedia subject associated with it. We're trying better organize Wikiversity so that top-level pages are either a course or full lesson on a particular subject. If you believe an Earning money redirect is necessary, what project should it redirect to? -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 12:33, 29 May 2015 (UTC)
:Maybe it should be a redirect to [[business]]?

== No Solicitation ==

Wikiversity's [[Wikiversity:Mission|mission]] is to create and host free learning materials and activities. Solicitation for external organizations and websites is not accepted. Let me know if you have any questions. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 22:30, 24 January 2016 (UTC)
: Can you site a policy specific about this? [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 22:45, 2 October 2016 (UTC)

::See [[Wikipedia:Wikipedia:External links#Links normally to be avoided]]. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 23:26, 2 October 2016 (UTC)
:::Is that referring to user pages or main namespace? Do Wikipedia policies apply to Wikiversity? Might wikiversity experience more growth if self promotion is allowed, at least on user pages? [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 23:45, 2 October 2016 (UTC)

::::Sorry, it comes from [[Wikiversity:External links]]. I clicked one too far before pasting the link. Regarding self promotion, see [[Wikiversity:Mission]]. The mission isn't growth or promotion elsewhere. The mission is free resources hosted here. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 00:33, 3 October 2016 (UTC)

== Edit Summary ==

Please include an Edit summary description with your changes. Also note that [[Wikiversity:Naming_conventions#Casing|Wikiversity naming convetions]] do not require Sentence case titles. In fact, most academics prefer Title Case. Let me know if you have any questions. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 16:37, 18 September 2016 (UTC)

== Open interviews ==

The pages created at [[Open interviews/I want to see aging defeated. Interview me!]] and [[Talk:Open interviews/I want to see aging defeated. Interview me!]] were self-promotion and solicitation for your own efforts outside of Wikiversity. They were not consistent with the [[Wikiversity:Mission]] and have been deleted. Please focus your editing efforts on developing learning projects that support the Wikiversity mission. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 13:44, 15 October 2016 (UTC)

:Wow. You really want to stifle growth on this site. Good job. My open interview was better received on Reddit. Please userify the page for a week so I can transwiki it to a different wiki. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 18:06, 15 October 2016 (UTC)

::I'm sorry to see that your vision of growth at Wikiversity is to grow Reddit at our expense. There is no need to restore the requested content. The text has been forwarded to your email account. Best of luck to you. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 19:48, 15 October 2016 (UTC)
:::How was that going to grow Reddit? Reddit has experienced enormous growth over the last few years. A large percentage of [https://www.reddit.com/r/TheoryOfReddit/comments/50xxj2/what_percentage_of_reddit_is_devoted_to_learning/ content on Reddit is directly and indirectly related to learning]. This site can learn a great deal from Reddit. I was attempting to emulate an extremely popular feature of Reddit here (which can be used for learning, research, and teaching), so that this site could potentially grow for the same excellent reason Reddit has related to learning - interviews (Reddit calls them AMA's Ask me anything). I hope you can figure out a way to be more welcoming to potential participants here on this site. Good luck. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 20:12, 15 October 2016 (UTC)

== Starting a journal ==

It's an interesting start you made at [[WikiJournal of Business and Economics]]. I've written some advice at: [[WikiJournal/Starting a journal]]. [[User:Mikael Häggström|Mikael Häggström]] ([[User talk:Mikael Häggström|discuss]] • [[Special:Contributions/Mikael Häggström|contribs]]) 17:00, 1 February 2017 (UTC)
:Thanks! [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 02:44, 2 February 2017 (UTC)
::Exciting to see interest in a new journal! Will articles be peer-reviewed before publication? <span class="nowrap">[[User:Evolution and evolvability|T.Shafee(Evo﹠Evo)]]<sup>[[User talk:Evolution and evolvability|talk]]</sup></span> 03:20, 4 March 2017 (UTC)
:::What do you think of the idea of forming it into a slightly broader journal of Social Science, to include e.g. Law, History and Sociology? Broader scope may make it easier to ensure sufficient content and interest to keep different WikiJournals maintainable. I suspect that WikiJournals will survive best if there are a small number (3-6) broad [[w:Discipline_(academia)|academic disciplines]]. I realise that it might make finding suitable external experts for peer review more difficult though. <span class="nowrap">[[User:Evolution and evolvability|T.Shafee(Evo﹠Evo)]]<sup>[[User talk:Evolution and evolvability|talk]]</sup></span> 08:26, 7 March 2017 (UTC)

== WikiJournal of Business and Economics at wikijournal.org ==

Hello Michael,

My name is Philip and I am a founder of wikijournal.org. I am involved to the group of users who works on different types of wikijournals like Wikijournal of Science, Wikijournal of Medicine etc. We are discussing merging wikijournal under one big broject - wikijournal.org. We discuss all possibilities and opportunities of the project here: https://en.wikiversity.org/wiki/Talk:WikiJournal/WikiJournal_User_Group#Developing_WikiJournal_project_at_wikijournal.org

There are also some discussions at wikijournal.org: https://en.wikijournal.org/wiki/Talk:WikiJournal

Feel free to join the group of users and register at wikijournal.org.

Kind wishes,

Philip

[[User:Fokebox|Fokebox]] ([[User talk:Fokebox|discuss]] • [[Special:Contributions/Fokebox|contribs]]) 14:02, 2 March 2017 (UTC)

== Inactive Participants ==

Please seek community input before listing additional users as active or inactive. There haven't been any active school contributors for a long time, but listing people as inactive has a different connotation than not listing participants at all. The community should have input on this. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 23:13, 30 September 2017 (UTC)
:OK. I will not use these templates anymore until community input is gathered. Feel free to gather the community input. I might too at some point in the future, before utilizing those templates and listing users and inactive. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 00:28, 1 October 2017 (UTC)

== acceptance of welcome ==

I am greatly pleased by your conduct of affairs here at wikiversity. I got into this to garner as much knowledge as possible to pass my courses in the university under computer science. Thank you. [[User:Jerusalem Sabbath|Jerusalem Sabbath]] ([[User talk:Jerusalem Sabbath|discuss]] • [[Special:Contributions/Jerusalem Sabbath|contribs]]) 06:11, 2 December 2017 (UTC)

== Share your experience and feedback as a Wikimedian in this global survey ==

<div class="mw-parser-output">
<div class="plainlinks mw-content-ltr" lang="en" dir="ltr">
Hello! The Wikimedia Foundation is asking for your feedback in a survey. We want to know how well we are supporting your work on and off wiki, and how we can change or improve things in the future. The opinions you share will directly affect the current and future work of the Wikimedia Foundation. You have been randomly selected to take this survey as we would like to hear from your Wikimedia community. The survey is available in various languages and will take between 20 and 40 minutes.

<big>'''[https://wikimedia.qualtrics.com/jfe/form/SV_5ABs6WwrDHzAeLr?aud=AE&prj=ot&edc=3&prjedc=ot3 Take the survey now!]'''</big>

You can find more information about this survey [[m:Special:MyLanguage/Community_Engagement_Insights/About_CE_Insights|on the project page]] and see how your feedback helps the Wikimedia Foundation support editors like you. This survey is hosted by a third-party service and governed by this [[:foundation:Community_Engagement_Insights_2018_Survey_Privacy_Statement|privacy statement]] (in English). Please visit our [[m:Special:MyLanguage/Community_Engagement_Insights/Frequently_asked_questions|frequently asked questions page]] to find more information about this survey. If you need additional help, or if you wish to opt-out of future communications about this survey, send an email through the EmailUser feature to [[:m:Special:EmailUser/WMF Surveys|WMF Surveys]] to remove you from the list.

Thank you!
</div> <span class="mw-content-ltr" dir="ltr">[[m:User:WMF Surveys|WMF Surveys]]</span>, 18:35, 29 March 2018 (UTC)

</div>


== Reminder: Share your feedback in this Wikimedia survey ==

<div class="mw-parser-output">
<div class="plainlinks mw-content-ltr" lang="en" dir="ltr">
Every response for this survey can help the Wikimedia Foundation improve your experience on the Wikimedia projects. So far, we have heard from just 29% of Wikimedia contributors. The survey is available in various languages and will take between 20 and 40 minutes to be completed. '''[https://wikimedia.qualtrics.com/jfe/form/SV_5ABs6WwrDHzAeLr?aud=AE&prj=ot&edc=3&prjedc=ot3 Take the survey now.]'''

If you have already taken the survey, we are sorry you've received this reminder. We have design the survey to make it impossible to identify which users have taken the survey, so we have to send reminders to everyone.
If you wish to opt-out of the next reminder or any other survey, send an email through EmailUser feature to [[:m:Special:EmailUser/WMF Surveys|WMF Surveys]]. You can also send any questions you have to this user email. [[m:Community_Engagement_Insights/About_CE_Insights|Learn more about this survey on the project page.]] This survey is hosted by a third-party service and governed by this Wikimedia Foundation [[:foundation:Community_Engagement_Insights_2018_Survey_Privacy_Statement|privacy statement]]. Thanks!
</div> <span class="mw-content-ltr" dir="ltr">[[m:User:WMF Surveys|WMF Surveys]]</span>, 01:33, 13 April 2018 (UTC)

</div>


== Your feedback matters: Final reminder to take the global Wikimedia survey ==

<div class="mw-parser-output">
<div class="plainlinks mw-content-ltr" lang="en" dir="ltr">
Hello! This is a final reminder that the Wikimedia Foundation survey will close on '''23 April, 2018 (07:00 UTC)'''. The survey is available in various languages and will take between 20 and 40 minutes. '''[https://wikimedia.qualtrics.com/jfe/form/SV_5ABs6WwrDHzAeLr?aud=AE&prj=ot&edc=3&prjedc=ot3 Take the survey now.]'''

'''If you already took the survey - thank you! We will not bother you again.''' We have designed the survey to make it impossible to identify which users have taken the survey, so we have to send reminders to everyone. To opt-out of future surveys, send an email through EmailUser feature to [[:m:Special:EmailUser/WMF Surveys|WMF Surveys]]. You can also send any questions you have to this user email. [[m:Community_Engagement_Insights/About_CE_Insights|Learn more about this survey on the project page.]] This survey is hosted by a third-party service and governed by this Wikimedia Foundation [[:foundation:Community_Engagement_Insights_2018_Survey_Privacy_Statement|privacy statement]].
</div> <span class="mw-content-ltr" dir="ltr">[[m:User:WMF Surveys|WMF Surveys]]</span>, 00:43, 20 April 2018 (UTC)

</div>


== Basic income cryptocurrency ==

Hi Michael Ten!

Your resource [[Basic income cryptocurrency]] appears to be well-developed and ready for learners! Would you like to have it announced on our Main Page News? --[[User:Marshallsumter|Marshallsumter]] ([[User talk:Marshallsumter|discuss]] • [[Special:Contributions/Marshallsumter|contribs]]) 02:06, 12 November 2018 (UTC)

== WikiJournal question ==

Hi Michael,

I noticed there's not been much movement on the [[Preliminary WikiJournal of Business and Economics]] that you initiated. Is that still in progress?

Also, please take a look at the Sister project proposal at [[meta:WikiJournal]] and comment/support/oppose.

[[User:Evolution and evolvability|T.Shafee(Evo﹠Evo)]]<sup>[[User talk:Evolution and evolvability|talk]]</sup> 04:23, 19 August 2019 (UTC)
: Everything on wiki's are in progress. Correct? Cheers and Limitless Peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 07:24, 20 August 2019 (UTC)
::Haha, true! Though I more meant is it progressing (as opposed to stalled). We're writing up a few applications at the moment for progressing the the journal group to a thematic organisation with and a full sister project platform for hosting ([[Talk:WikiJournal User Group#Sister project and Thematic organisation applications|link]]), so I'm looking to know whether it's useful to include it in the applications. [[User:Evolution and evolvability|T.Shafee(Evo﹠Evo)]]<sup>[[User talk:Evolution and evolvability|talk]]</sup> 09:33, 20 August 2019 (UTC)
:::On quasi pause, ready for reactivation potentially, might properly describe its status. Hmmm. Cheers and Limitless Peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 00:05, 21 August 2019 (UTC)

== Banking the unbanked ==

Hi Michael Ten!

Your resource [[Banking the unbanked]] appears to be well-developed and ready for learners! Would you like to have it announced on our Main Page News? --[[User:Marshallsumter|Marshallsumter]] ([[User talk:Marshallsumter|discuss]] • [[Special:Contributions/Marshallsumter|contribs]]) 02:37, 4 September 2019 (UTC)
:Seems good! Cielo Milagros. 🌞

== Draft Redirects ==

Content in the Draft namespace has been deemed as "not yet ready for main space listing". Redirects to Draft namespace from main space are not accepted. In this case, it is also a violation of the RFD discussion on Main Page "lectures". Please avoid redirects to Draft space in the future. -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 13:25, 26 October 2019 (UTC)

== Wikipedia Readings ==
I see that you created a [https://en.wikiversity.org/w/index.php?title=Drones%2FAgriculture&type=revision&diff=2280213&oldid=2121592 section "Wikipeadia Readings"] and moved that from "See also" to "Wikipedia Readings". Makes a lot of sense to me to do that in general. Would you recommend that for all learning resources in Wikiversity, even if there is just one resource?--[[User:Bert Niehaus|Bert Niehaus]] ([[User talk:Bert Niehaus|discuss]] • [[Special:Contributions/Bert Niehaus|contribs]]) 07:02, 26 April 2021 (UTC)
::If there is just one link to Wikipedia? It seems maybe noting that it links to Wikipedia rather than Wikiversity might be ideal. Somehow, if it is not otherwise noted. Hmmmm. Cheers and serenity. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 03:41, 27 April 2021 (UTC)

== Nocat ==

Draft resources are not categorized. Users restricted to Draft: space have tried to use this approach to bypass their restrictions.

Also, please note that adding {{Tlx|Nocat}} makes it more difficult for others to add categories in the future. Nocat can't be removed using [[mw:Manual:HotCat|HotCat]], and almost everyone who does significant category work at Wikiversity uses this tool. Please consider using [[Special:UncategorizedPages]] instead to identify pages needing categories, and add categories rather than adding Nocat.

Thanks! -- [[User:Dave Braunschweig|Dave Braunschweig]] ([[User talk:Dave Braunschweig|discuss]] • [[Special:Contributions/Dave Braunschweig|contribs]]) 21:00, 1 May 2021 (UTC)
: No problem. OK. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 23:23, 1 May 2021 (UTC)

== Creativity course merger ==

I've noticed that you and I have created our own creativity courses, and would love to see if you could consider us merging the two courses? I think that we could make this work, Michael. [[User:Contributor 118,784|<b style="color:#070">Contributor</b><sup style="color:#707">118,784</sup>]] [[User talk:Contributor 118,784|<span style="color:#00F">''Let's talk''</span>]] 11:16, 27 October 2023 (UTC)
: Links the two courses you are referring to? Thanks and limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 03:38, 5 November 2023 (UTC)
::[[Creativity Introduction]] [[User:Contributor 118,784|<b style="color:#070">Contributor</b><sup style="color:#707">118,784</sup>]] [[User talk:Contributor 118,784|<span style="color:#00F">''Let's talk''</span>]] 09:16, 5 November 2023 (UTC)
::: What is the page of the one I created? thanks for reminding me. Limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 02:36, 8 November 2023 (UTC)
::::That would be [[Creativity]]. Limitless peace to you too! [[User:Contributor 118,784|<b style="color:#070">Contributor</b><sup style="color:#707">118,784</sup>]] [[User talk:Contributor 118,784|<span style="color:#00F">''Let's talk''</span>]] 08:38, 8 November 2023 (UTC)

== An article you edited is proposed for deletion ==

See [[Nation]] [[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 05:46, 2 January 2024 (UTC)

== Invitation to discuss page deletion policy ==

A discussion that might interest you has been started at [[Wikiversity:Requests_for_Deletion#Wikiversity:Deletion_Convention_2024]]. -- [[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 18:18, 15 February 2024 (UTC)

== Suggested changes for draftify ==

I pinged you from another page before realizing the question belonged on your talk page: I suggest two changes to draftify. Let me know if you want me to edit Template:Draftify to implement these changes. See ''[[User:Guy_vandegrift/sandbox]]''.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 00:55, 16 March 2024 (UTC)

== Two items related to Draftify ==

'''Item A:''' We seem to have polling places all over Wikiversity:
#[[Wikiversity_talk:Drafts#policy_and_page_change_suggestion]]
#[[Wikiversity_talk:Deletions#Proposed_modifications]]
#[[Wikiversity:Colloquium#Please_vote_on_whether_to_allow_pages_in_draftspace_to_remain_indefinitely]]

Would you like me to place notices on two of them to vote on the third? If, so I suppose the Colloquium is the best choice. Do you agree?

'''Item B:''' I have some ideas about changing your Draftify template. See details at [[User:Guy vandegrift/sandbox]]

{{ping|Jtneill|MathXplore}} Yours truly --[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 15:36, 16 March 2024 (UTC)
::I do not have a strong preference how it is geared. (One of) My goal(s) is to help ensure good faith Creative Commons contributions (that are not blatant spam or so forth) at least remain in draft namespace indefinitely. I have no strong preference where this is voted on or how this is accomplished. Several paths may lead to the top of Mt. Fuji/Rome. limitless peace [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 23:41, 16 March 2024 (UTC)
:::I will go ahead and change the template: It's not a controversial edit. And I will post a note on the colloquium abo--ut the three places where the draftspace motion is being discussed. I will look for you on Mt. Fugi.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 01:36, 17 March 2024 (UTC)

==Changing support from draftspace to draft archive==
A lot of deletion requests are stuck at a near consensus. My guess is that you would support moving all of the articles into [[Draft:Archive/Foobar]]. If you go to [[Draft:Archive]] you will see that I have a search feature that might make it a more desirable location than [[Draft:Foobar]]. If you OK this on your talk page, I can insert a permalink to this conversation. That would save you the trouble of going to each section and writing something. The pages will be easy to find there, and we can move them back to draftspace or mainspace at any time in the future.
{{colbegin}}
*Wikiversity:Requests for Deletion#Space fleet academy

*Wikiversity:Requests for Deletion#Openness

*Wikiversity:Requests for Deletion#OpenOffice.org

*Requests for Deletion#Metadata

*Wikiversity:Requests for Deletion#Facilitation

*Wikiversity:Requests for Deletion#Space fleet academy

*Wikiversity:Requests for Deletion#

*Wikiversity:Requests for Deletion#Facilitation
{{colend}}

Happy editing, --[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 16:03, 21 March 2024 (UTC)
:I appreciate the fyi. My support for moving to draft name is just that - support for moving to draft namespace. i do not have time today to clarify my support (if you mean related to organizational structure) further - although i noted some views on that somewhere but i would have to find those. I am not sure when I will have more time to volunteer editing this wiki. maybe tomorrow. maybe in two weeks. although I will try to check notifications and such sooner if possible. limitless peace. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 00:06, 23 March 2024 (UTC)
:With that said, if pages in Draft namespace will be kept indefinitely, I do think that Draft:PageName is a better longer term organizational structure than Draft:Archive/PageName. I would support a separate Archive: namespace if there is a justification/reason like if there are things that should be archived (that were good faith Creative Commons content contributions that are education/teaching/research/learning related that could be useful or valuable but not belong in draft nor main namespace). I also support there being various organizational pages within the draft namespace so that draft content could be found in order to be developed further and spark new ideas for new learning/teaching/research/educational content. Although opinions and views on this may vary. I support diversity of thought. [[User:Michael Ten|Michael Ten]] ([[User talk:Michael Ten|discuss]] • [[Special:Contributions/Michael Ten|contribs]]) 00:30, 23 March 2024 (UTC)
::I definitely agree there, Michael. I am not a fan of throwing away freely licensed good-faith contributions. Whoever is interested to learn about something should be able to read it. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 21:05, 11 April 2024 (UTC)
:I also see no reason for deleting draft-space articles that do no harm: Wikiversity can't delete itself into a better wiki. But a majority of active cleanup editors don't see it that way. I am looking for a compromise. I think "quality" control should happen only at the top of mainspace, with the editors of that top page choosing how to organize the subpages. I am neutral about top namespace stubs, but see no reason to delete them or disallow others to write them. But the sentiment among the active cleanup editors is trending the other way. My plan is to move pages that might have merit into Draft:Archive and ignore what the others do. I like the way you write.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 00:56, 23 March 2024 (UTC)

Backup/Causes of data loss

2024-03-20T22:39:23Z

Elominius: + reference

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref><ref>{{cite AV media |url=https://archive.org:443/details/youtube-NtxPrIFpDsk |author=éclairJess |date=2018-08-30 |language=de |access-date=2024-03-20 |title=WICHTIG !!EINSELF! (wird später gelöscht) |website=Internet Archive }} (originally uploaded to YouTube) - Artist explains she accidentally lost a partition containing her works.</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Wikiversity talk:Custodianship

2024-03-17T00:47:56Z

Elominius: support

{{TOC right}}

<big>Please see the archives for prior discussions:</big>
* [[/Archive 1/]] 2006 - 2008
* [[/Archive 2/]] 2009 - 2010
* [[/Archive 3/]] 2011
* [[/Archive 4/]] 2015
* [[/Archive 5/]] 2016
* [[/Archive 6/]] 2018 - 2019

== Proposal to allow custodians to use mass-delete ==

At [[special:permalink/2609683#Does_anybody_know_how_to_delete_all_pages_by_a_single_user?]], there was a related discussion about this matter. I brought the agenda to the colloquium for the community's attention, and I was suggested to start a proposal here ([[special:diff/2610994]]). As can be seen at [[Special:ListGroupRights]], only bureaucrats are allowed to use mass-delete under current settings, but many Wikimedia projects allow this to admins (equal to our custodians). Please note that global sysops can also use mass-delete. What does our community think about this? Should we keep the current settings, or should we grant mass-delete to our custodians as a new standard? [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 07:12, 8 March 2024 (UTC)
: Curious - is there is a list somewhere of what permissions are provided on which WMF projects? -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 22:43, 8 March 2024 (UTC)
:: [[Special:ListGroupRights]] is available at all WMF projects. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:17, 9 March 2024 (UTC)
:: Please also note that [[:mw:Extension:Nuke]] and [[:b:MediaWiki_Administrator%27s_Handbook/Page_Deletion#Mass_page_deletion]] assumes that admins can handle mass-delete. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:24, 9 March 2024 (UTC)
:a) I'm curious if there's a conflict of interest seeing that [[Wikiversity:Candidates for Custodianship/MathXplore|there is a proposal to make you custodian]] (whatever that means), and b) What's mass-delete? [[User:Username142857|Username142857]] ([[User talk:Username142857|discuss]] • [[Special:Contributions/Username142857|contribs]]) 14:31, 9 March 2024 (UTC)
::It's not a conflict of interest, but the two issues are certainly linked. MathXplore was nominated because they have considerable administrative experience with other Wikis and because they are currently actively involved with Wikversity "cleanup". MathXplore's interest in mass-deleting was in response to my request for information on that action. I am not aware that MathXplore ever asked to be nominated, but if that was the case, I fully support it. A number of editors are interested in removing low-quality pages (or at least moving them to less conspicuous locations.) I am somewhat of a bottleneck to that process because as a custodian, I am one of the few people who can delete pages (see '''[[Wikiversity:Deletions]]'''.) Although there is no conflict of interest, there is a connection between these two requests: MathXplore is less prone to mistakes than I. Giving MathXplore custodianship and giving custodians mass-deletion would speed up this "cleanup" process. I have never see MathXplore display poor judgement, and that is why I support giving him both. Only the adoption of both proposals would allow MathXplore to delete a large number of pages without anybody's approval.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 15:39, 9 March 2024 (UTC)
:::Thanks @[[User:Username142857|Username142857]] and @[[User:Guy vandegrift|Guy vandegrift]] for pointing out that there is a connection between: (a) this proposal by MathXplore to allow custodians to use mass-delete and (b) the [[Wikiversity:Candidates for Custodianship/MathXplore|nomination (by myself) for MathXplore to move from curator to custodian]]. As Guy points out, it is not necessarily a conflict of interest, but it is important that the connection be pointed out in case anyone has concerns they would like to raise. -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 23:03, 9 March 2024 (UTC)
:::: Thank you for your responses. I noticed that we don't have a COI-related policy. Until formal agreements, [[:w:Wikipedia:Conflict of interest]] can be used as our reference. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:03, 10 March 2024 (UTC)
::::: {{ping|Guy vandegrift}} I declare that I never contacted anyone else about custodianship before [[special:permalink/2611262#Custodianship]]. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:05, 10 March 2024 (UTC)
::::::{{ping|MathXplore}} I'm not accustomed to doing things by the book. Am I supposed to refrain from voting on this because I do have a clear benefit if it passes? My plan was to be one of the last to vote.--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 03:23, 10 March 2024 (UTC)
::::::: Voting or not is all up to you. I'm not sure if you have benefit because the deletions that you are handling are not suitable for mass-delete. They come from different users, and they are neither vandalism nor illegal. Mass-delete aims to accelerate deletion of vandalism etc. by a single user. In the past, I have seen many discussions about admin configurations or deletion policy at other WMF projects (recently I have seen [[:q:special:permalink/3478221#Flood_flag]]), but admins were allowed to join the discussion as one user. So I'm not suprised to see you at here. We are a smaller community (and a GS wiki), if custodians don't join the discussion, then others won't. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 06:16, 10 March 2024 (UTC)
::::::::What does 'illegal' mean in this context, and why couldn't someone just make a resource titled 'Wikiversity: Conflict of interest' to create a COI-related policy? Also, how is this project different from others and what makes a difference meaningful as opposed to not meaningful (examples of both would be nice)? Sorry if I'm asking too many questions :) [[User:Username142857|Username142857]] ([[User talk:Username142857|discuss]] • [[Special:Contributions/Username142857|contribs]]) 15:01, 12 March 2024 (UTC)
::::::::: Examples of illegal content include (but maybe not limited to) copyright violations, privacy violations, serious derogatory, etc. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:04, 13 March 2024 (UTC)
::::::::: Again, we are a smaller community. We don't have enough workforce to make every kind of rules, and making rules is not are main scope (please see [[Wikiversity:Scope]]). You are welcome to make drafts of COI-related policy, but promotion to official policy will need another discussion. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:08, 13 March 2024 (UTC)
::::::::: Unlike others, we are an educational project. Things without educational objectives can disappear ([[WV:CSD]]). Wikipedia is an encyclopedia and may host school projects, but not exactly educational. There are many non-educational content over there, such as video games, movies, animation articles without any educational viewpoints. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:14, 13 March 2024 (UTC)
::::::::: For examples of meaningful differences at Wikiversity, I know [[WikiJournal]] (preprints) and [[Wikidebate]]. WikiJournals can be educational (especially for higher grades), but since they will include original research, they cannot be accepted at Wikipedia ([[:w:Wikipedia:No original research]], [[:w:Wikipedia:What_Wikipedia_is_not#Wikipedia_is_not_a_manual,_guidebook,_textbook,_or_scientific_journal]]). Wikidebates can help people learn the subject in general or how to debate, but since Wikipedia disallows forums for subjects in general, it cannot exist over there ([[:w:Wikipedia:What_Wikipedia_is_not#Wikipedia_is_not_a_publisher_of_original_thought]]). [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:22, 13 March 2024 (UTC)
::::::::: An example of possibly non-meaningful differences at Wikiversity was the collection namespace ([[special:permalink/2612077#RFC:_Deprecate_and_remove_the_Collection:_namespace.]]). In the beginning, it was believed to be meaningful, but it was found that it was not used efficiently. Another example of possibly non-meaningful differences at here is the issue that we are facing at here, limiting mass-delete to bureaucrats when more than half of our bureaucrats are inactive. This restriction is affecting the community's ability to handle vandalism quickly, and has nothing to do with Wikiversity's educational objective. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:33, 13 March 2024 (UTC)
:: The first part of your question has been answered by our current custodians, I don't think I have to add anything else. Technical details of mass-delete are provided at [[:mw:Extension:Nuke]]. It can be used by selecting a specific user, and then deleting pages created by them at once. The scope of this feature is to remove obvious vandalism or illegal content (copyright violations, attack pages, etc.) faster. A recent example of using mass-delete is [[Special:Contributions/39.50.199.52]] (already reported to [[WV:RCA]]). As I pointed out above, the Wikimedia standard is to allow mass-delete to admins (they are known as [[Wikiversity:Custodianship|custodians]] in our project). If we are going to make a difference, then those who made them should be able to explain them. I have recently asked why we limited mass-delete to bureaucrats ([[special:diff/2609059]]) and the answer was [[special:diff/2609060]]. We are different from other projects. Meaningful differences should be saved, but questionable differences should be corrected (in this case, restore standard technical settings). We don't need to make differences to be different. This is the background of this proposal. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:01, 10 March 2024 (UTC)

; Voting
: {{support}} It seems that most of the clean-up work at en.wv is currently being performed by sysops/custodians and if they are requesting access to [[Special:Nuke]] functionality to help them do that work, I'm keen to support that. Sincerely, James -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 22:43, 8 March 2024 (UTC)

: {{support}} There seems to be a lot of content generated and a lot of content left incomplete and dormant. Probably there's a lot to maintain and I'm all for making it easier on whomever does it. [[User:Addemf|Addemf]] ([[User talk:Addemf|discuss]] • [[Special:Contributions/Addemf|contribs]]) 02:04, 9 March 2024 (UTC)

: {{support}} I've run into this problem where I've wanted to nuke a vandal's contributions but wasn't able to - manually deleting several pages is a hassle. This would be a great additional to our toolkit. —[[User:Atcovi|Atcovi]] [[User talk:Atcovi|(Talk]] - [[Special:Contributions/Atcovi|Contribs)]] 17:00, 11 March 2024 (UTC)
:{{support}} Safe enough to try, as I believe I wrote on a previous thread. —[[User:Koavf|Justin (<span style="color:grey">ko'''a'''vf</span>)]]<span style="color:red">❤[[User talk:Koavf|T]]☮[[Special:Contributions/Koavf|C]]☺[[Special:Emailuser/Koavf|M]]☯</span> 17:40, 11 March 2024 (UTC)

:{{support}} - Would improve response times and any damage from incorrect use is reversible by a custodian anyway. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 00:47, 17 March 2024 (UTC)

Wikiversity:Candidates for Custodianship/MathXplore

2024-03-17T00:45:10Z

Elominius: /* Comments */ Go for it!

==={{User|MathXplore}}===
I'd like to nominate MathXplore (who has been a curator since Jan 29 2023) as a custodian. This wiki is in need of more active custodians. MathXplore has been one of a small handful of curators/custodians doing most of the maintenance work (from what I can tell) recently. They obviously know their way around a wiki, with very good technical knowledge and a level head in communicating and decision-making. I think the community can trust and benefit from MathXplore having custodial tools. @[[User:MathXplore]] feel free to add anything else you think is relevant. Sincerely, James -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 03:38, 9 March 2024 (UTC)
: Nomination was accepted at [[special:permalink/2611262#Custodianship]]. For previous experiences with advanced permissions, I have global rollback ([[:m:Steward_requests/Global_permissions/2023-03#Global_rollback_for_MathXplore]]), Simple English Wiktionary admin ([[:wikt:simple:special:permalink/516047#Requests_for_adminship]]), and English Wikiversity curator. I look forward to helping with non-controversial (revision) deletion tasks (but also handling refunding requests if applicable) and checking deleted contributions to identify xwiki or long-term abuses. I appreciate your consideration. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 07:34, 9 March 2024 (UTC)
::What's global rollback and why is there a Simple English Wikipedia and Wiktionary, but no, for example, *Simple Italian (for example) Wikipedia and Wiktionary? [[User:Username142857|Username142857]] ([[User talk:Username142857|discuss]] • [[Special:Contributions/Username142857|contribs]]) 15:13, 12 March 2024 (UTC)
::: Global rollback is the ability to use rollback at all Wikimedia projects. Please see [[:m:Global rollback]] for their guidelines and policies. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:39, 13 March 2024 (UTC)
::: We have Simple English Wikipedia (simplewiki) and Simple English Wiktionary because there is an agreement to keep these projects open. For simplewiki we have [[:m:Proposals for closing projects/Closure of Simple English Wikipedia]], [[:m:Proposals for closing projects/Closure of Simple English Wikipedia (2)]], and [[:m:Proposals for closing projects/Closure of Simple English Wikipedia (3)]]. For simplewikt, please see [[:m:Proposals for closing projects/Closure of Simple English Wiktionary]], [[:m:Proposals for closing projects/Closure of Simple English Wiktionary (2)]], and [[:m:Proposals for closing projects/Closure of Simple English Wiktionary (3)]]. In general, opening or closing a Wikimedia project requires global agreements. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:54, 13 March 2024 (UTC)
::: For simple Italian, I do not recognize any valid proposals or agreements to open them (no page found on Meta-Wiki). Situation is similar for other languages, the proposals were rejected ([[:m:Requests for new languages/Wikipedia Simple German 4]], [[:m:Requests for new languages/Wikipedia Simple Spanish 4]], [[:m:Requests for new languages/Wikipedia Simple Latin]]). [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:00, 13 March 2024 (UTC)

==== Questions ====
What's a curator? What's a custodian? [[User:Username142857|Username142857]] ([[User talk:Username142857|discuss]] • [[Special:Contributions/Username142857|contribs]]) 14:28, 9 March 2024 (UTC)

:See [[Wikiversity:Support staff]] (which has links to [[Wikiversity:Custodianship]] and [[Wikiversity:Curators]].)--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 20:21, 9 March 2024 (UTC)

==== Questions for the candidate ====
# Your Userpage says you have a native understanding of both English and Japanese. I don't remember ever seeing anyone else who displays such an unusual claim. Would you please explain how you achieved this distinction? Thanks in advance, [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 00:05, 11 March 2024 (UTC)
#: I just studied harder than others. Please note that nobody is allowed to skip English lessons in my locality ([[:w:English-language education in Japan]]). I had more English lessons than Japanese literature classes. In university, many textbooks were written in English. I'm not surprised to see bilingual speakers, and I have met various bilingual users (including those with different language combinations) at Wikimedia projects. In my opinion, being bilingual is neither unusual nor extraordinary. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:47, 11 March 2024 (UTC)
#::Unless you grew up with both languages, surely at least one of them will be a second language? Also, why would Japanese (I assume) universities use textbooks written in a foreign language? [[User:Username142857|Username142857]] ([[User talk:Username142857|discuss]] • [[Special:Contributions/Username142857|contribs]]) 15:20, 12 March 2024 (UTC)
#::: I have been studying English since kindergarten. In other words, I have grown up with both languages. I believe that people can use 2 languages (or maybe more) fluently just like they are using 2 hands in their life. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:41, 13 March 2024 (UTC)
#::: Please note that there are not enough Japanese textbooks for all academic subjects (especiallly those for higher grades). If there are no available Japanese textbooks, then using English textbooks will be an alternative option. In addition, some universities may have different language preferences (such as international universities, classes including foreign students, classes taught by non-Japanese teachers, etc.), if there is any reason to use English, then the textbook may be English. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:45, 13 March 2024 (UTC)
# How responsive do you plan to be to members of wikiversity. I am curious because I have tried to communicate with [https://en.wikiquote.org/w/index.php?title=User_talk:MathXplore&oldid=3413229#I_undid_one_of_your_many_category_ediTS you on your talkpage at Wikiquote] and found you be "the strong and silent type". I am curious to find out if it is only me, or is this your normal mode of communication. [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 00:28, 11 March 2024 (UTC)
#: At all Wikimedia projects where I'm active (including Wikiversity), I always try to provide an answer within a reasonable amount of time, just like I answered your previous question above. I also plan to do so in the future at Wikiversity. On the other hand, I sometimes try to make my responses shorter to avoid [[:w:Wikipedia:Wall of text]] and make things easier to understand. I apologize if my previous responses did not satisfy you. Please let me know if I have answered your question or not. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:53, 11 March 2024 (UTC)
#::Thank you for asking, yes you have answered my question. I would like to add that even thought this is the first time I am participating in a wikiversity ''Candidates for Custodianship'' I have participated in similar discussions on the WikiMedia-projects, so have no idea what to expect. For example it is not clear to me:
#::* When is this discussion expected to close?
#::* Is there a limit to the number of question each participant may ask?
#::* Who will make the final decision
#::* Will the decision be based on a numerical count of votes, or will it be base on [[Wikiversity:Consensus]].
#::I also have some off-topic comments regarding your answers here, but I will share those with you on your user-talk-page if you are interested. [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 22:05, 13 March 2024 (UTC)
#::: (About discussion closure date) According to [[Wikiversity:Custodianship#How_does_one_become_a_custodian?]], "After one week of evaluation, a bureaucrat will make the final decision based on the arguments provided in the discussion." ([[special:permalink/2608768#How_does_one_become_a_custodian?]]). In other words, the CFC will end after one week from the start of the discussion. In general, many discussions about advanced permissions (including adminship) require one week for discussion. While [[Wikiversity:Custodianship]] seems to be partially outdated (referring to probationary custodians that no longer exists at here), I believe this point remains valid. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:01, 14 March 2024 (UTC)
#::: (About question number limits) I have not seen any documented limit to the number of questions each participant may ask at [[WV:CFC]] or [[Wikiversity:Custodianship]]. Maybe we don't have such a rule. Anyone is welcome to provide information if I have missed anything. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:04, 14 March 2024 (UTC)
#::: (About the discussion closer) According to [[Wikiversity:Custodianship#How_does_one_become_a_custodian?]], "After one week of evaluation, a bureaucrat will make the final decision based on the arguments provided in the discussion." So the discussion closer will be our bureaucrats (please see [[Special:ListUsers/bureaucrat]] and [[Wikiversity:Bureaucratship]]). At all Wikimedia projects, only bureaucrats can make non-admins to admins. In other words, only bureaucrats can enforce the results of RFAs. Usually, a bureaucrat who has not been involved in the voting will close the discussion. On the other hand, we have only 3 bureaucrats and 2 of them are not active. Please see [[special:permalink/2530298]] for a Wikibreak declaration from one of our bureaucrats. Please also note that the currently most active Wikiversity bureaucrat is the nominator of this discussion. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:11, 14 March 2024 (UTC)
#::: (About the basis of the final decision) The latest instructions at [[WV:CFC]] do not mention any number of votes. I think it is reasonable to expect decisions based on [[Wikiversity:Consensus]]. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:14, 14 March 2024 (UTC)
#::: (For any other comments beyond the scope of this CFC) You are welcome to visit my user talk page, or you can use [[Wikiversity:Colloquium]] if your comments need the community's attention. Please note that once the CFC has been closed, the CFC page may be (fully) protected to save the whole discussion. If you have anything else to say after the CFC, please consider using my user talk page, the colloquium, or any other relevant discussion venue. If there has been any failure to answer your questions, please let me know as soon as possible. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:22, 14 March 2024 (UTC)
# How will being a Custodian change your daily schedule. You are already contributing thousands of edits a month to many wmf-wikis, so how will you manage the load of your new responsibility? How will you deal with the side-effects of being more visible, such as being the target of unhappy community members who use colorful language (and more). I believe you are currently an administrator on one wiki, but I think(?) the simple-wiktionary is smaller than en-wikiversity, do you plan to continue there as well? [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 16:21, 14 March 2024 (UTC)
#: (About daily schedule) As curator, I always have to request custodian action ([[WV:RCA]]) when there is something that I cannot do, and wait until the arrival of our custodians. Being a custodian means I may not need to wait for them. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:57, 15 March 2024 (UTC)
#: (About activities outside Wikiversity) I indeed have many edits at many WMF-wikis every month. On the other hand, many of those edits are semi-automated (Please see [[:w:WP:TW]], [[:w:WP:HC]], [[:w:simple:WP:TW]], [[:m:SWViewer]], [[:commons:Help:Gadget-Cat-a-lot]], [[:m:User:Syunsyunminmin/SReporter.js]], and [[:m:User:Xiplus/TwinkleGlobal]] for the tools that I'm using). In other words, the many edits outside Wikiversity do not require an enormous amount of time, so the impact on my Wikiversity activity status is limited. Therefore, I do not see any obstacles to my Wikiversity participation. If needed, I can close some non-Wikiversity tabs and concentrate here. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:08, 15 March 2024 (UTC)
#: (About how to deal with unhappy community members) It depends on how they are unhappy. If they are unhappy with non-controversial simpler things (such as those listed at [[:category:Wikiversity FAQ]]), I might be able to handle them. If they're unhappy with complicated or controversial things (for example, somebody is unhappy with something but others say they are OK with that), they should be handled by multiple custodians/curators or the community rather than myself. If they're unhappy with things I don't understand, then I will find somebody who knows better than me (for example, there are some Wikiversity subprojects in which I have limited involvement, such as [[Wikidebate]] and [[WikiJournal]], and things like these may happen in these areas). [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:24, 15 March 2024 (UTC)
#: (About colorful languages) We have our policy ([[Wikiversity:Civility]]), so our reactions should be based on this documentation. For obvious and serious colorful languages, [[WV:REVDEL]] will be applied quickly. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:27, 15 March 2024 (UTC)
#: (About Simple English Wiktionary) I have neither declared nor decided to quit any WMF projects. I plan to continue Simple English Wiktionary contributions as long as possible. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:29, 15 March 2024 (UTC)
# You can already delete pages as a Curator. What will you be able to do in addition as a Custodian and how do you plan to use those abilities? [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 16:21, 14 March 2024 (UTC)
#: (About deletions) Indeed, I can already delete pages as a curator ([[WV:CUR/D]]). In other words, curators can only delete pages, they cannot delete revisions and logged records. Custodians can also handle revision deletions ([[WV:REVDEL]]), and I plan to handle these as instructed. For contentious cases, I will discuss them with other custodians. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:34, 15 March 2024 (UTC)
#: (Editorial restrictions) These are only allowed to custodians ([[WV:CUST/B]]), and [[Wikiversity:Blocking policy]] will be applied. Again, for any kind of contentious cases, I will discuss them with other custodians. For those who are very well-known for evading blocks and/or [[:m:GL|global locks]] (such users can also be found via [[:w:WP:LTA]] and [[:w:WP:SPI]] archives, etc.), I believe I know what to do. I have a history of reporting such users to [[WV:RCA]] (https://xtools.wmcloud.org/topedits/en.wikiversity.org/MathXplore/4/Request%20custodian%20action) and [[:m:SRG]] (https://xtools.wmcloud.org/topedits/meta.wikimedia.org/MathXplore/0/Steward%20requests/Global). I think these can explain my policy understanding. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:47, 15 March 2024 (UTC)
#:: (Additional info for editorial restrictions) We are smaller than enwiki, and we have fewer vandals than them. Blocks are sometimes needed, but I don't think they will be needed everyday like enwiki. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:50, 15 March 2024 (UTC)
#:: (About IP blocks) In the past I have requested various IP blocks including IP range blocks ([[:w:Wikipedia:Blocking_IP_addresses#Range_blocks]]) at [[WV:RCA]], [[:m:SRG]], etc. but I don't have absolute certainty about how long the block should be, and how the block range should be set. I have nothing more than generic understanding (start the block with shorter term, extend blocks when persistent, etc.) and recently learned about [[:w:en:User:TonyBallioni/Just block the /64]]. For anything I'm not sure about IP blocks, I will discuss with other custodians or anyone else who has more knowledge than me (but for ongoing violent vandalism cases, I may consider to activate blocks with the minimal term and range). Checking IP block settings by other admins in other projects might be a useful reference when handling open proxies ([[:m:No open proxies]]). If applicable, directly reporting IPs to [[:m:SRG]] instead local actions can be accurate than my decisions. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:17, 15 March 2024 (UTC)
#: (MediaWiki namespace edits) These are only allowed to custodians, etc. ([[WV:CUST/MW]]). The most common one that I see is the update of [[MediaWiki:Sitenotice]], just like it was updated for the CFC announcement. For non-controversial MediaWiki:Sitenotice edit requests, I look forward to handling them. As mentioned above, if there is anything unclear to me, I will ask questions to other custodians before any actions. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 02:54, 15 March 2024 (UTC)
#:: (About the frequency of MediaWiki namespace edits) In my opinion, the frequency of MediaWiki namespace edits is very lower than block requests and deletions. I don't imagine that I will be handling such requests in every week, but I will try to check if there is anything that I may need to help. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:22, 15 March 2024 (UTC)
#:: (About non-custodian MediaWiki namespace edits) I forgot to mention this in above. [[:m:Stewards]], [[:m:global sysops]], [[:m:global interface editors]], and [[Wikiversity:Interface administrators]] can also do this. Unless those causing obvious and serious errors, I have no plan to reverse or override their actions. They are users that have technical experiences that I don't have, I believe their actions can be trusted in many cases. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:29, 15 March 2024 (UTC)
#: (Others) In above, I have answered about common custodian actions that I previously requested to [[WV:RCA]]. For a complete list, please see [[Special:ListGroupRights#sysop]]. There are many uncommon custodian privileges included there, and I don't think I will be using them in a near future. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 03:01, 15 March 2024 (UTC)
# Do you plan on obtaining other so-called hats shortly? [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 16:21, 14 March 2024 (UTC)
#: Maybe yes, but I have not decided when to do so. Please see [[:w:simple:special:permalink/9412939#RfA]] for an example. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:46, 15 March 2024 (UTC)
# What in your opinion are the most pressing needs of en-wikiversity in general? [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 16:21, 14 March 2024 (UTC)
#: I think additional vigilence is needed for the detection of unnoticed issues. As curator, I have already deleted many pages that remained for years despite clearly fitting to [[WV:CSD]]. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 01:49, 15 March 2024 (UTC)

==== Comments ====
* {{Support}} - As nominator. -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 03:38, 9 March 2024 (UTC)
* {{Support}} - Everything I've seen of them seems good to me! [[User:Addemf|Addemf]] ([[User talk:Addemf|discuss]] • [[Special:Contributions/Addemf|contribs]]) 04:48, 9 March 2024 (UTC)
* {{Support}} Recently active and knowledgeable editor who is easy to work with--[[User:Guy vandegrift|Guy vandegrift]] ([[User talk:Guy vandegrift|discuss]] • [[Special:Contributions/Guy vandegrift|contribs]]) 04:51, 9 March 2024 (UTC)
* {{support}} Have seen lots of good work from them here and elsewhere —[[User:94rain|94rain]] ([[User talk:94rain|discuss]] • [[Special:Contributions/94rain|contribs]]) 06:08, 9 March 2024 (UTC)
* {{support}} —[[User:Atcovi|Atcovi]] [[User talk:Atcovi|(Talk]] - [[Special:Contributions/Atcovi|Contribs)]] 12:03, 11 March 2024 (UTC)
*{{Support}} Super-uncontroversial. —[[User:Koavf|Justin (<span style="color:grey">ko'''a'''vf</span>)]]<span style="color:red">❤[[User talk:Koavf|T]]☮[[Special:Contributions/Koavf|C]]☺[[Special:Emailuser/Koavf|M]]☯</span> 17:21, 11 March 2024 (UTC)
*{{support}} as they're a trusted user doing anti-vandalism activities. [[User:Codename Noreste|Codename Noreste]] ([[User talk:Codename Noreste|discuss]] • [[Special:Contributions/Codename Noreste|contribs]]) 01:17, 12 March 2024 (UTC)
*{{Support}} Excellent and prolific contributions. --[[User:Lbeaumont|Lbeaumont]] ([[User talk:Lbeaumont|discuss]] • [[Special:Contributions/Lbeaumont|contribs]]) 16:10, 13 March 2024 (UTC)
* {{support}} has answered questions here generously [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 16:21, 14 March 2024 (UTC)
* {{abstain}} I abstain, not because of MathXplore, but because I reminded myself of the need to create a good desysopping policy. A good desysopping policy is one in which a superminority (33.3% + 1) can remove an administrator from the function if it so wishes. To move things toward that policy, I might even oppose an admin nomination that contains no clause for easy desysopping, but my experience from the English Wiktionary was that it annoyed a great deal of editors, so I merely formally abstain. --[[User:Dan Polansky|Dan Polansky]] ([[User talk:Dan Polansky|discuss]] • [[Special:Contributions/Dan Polansky|contribs]]) 10:32, 15 March 2024 (UTC)
*:@[[User:Dan Polansky|Dan Polansky]], This is a great point. I assume there is a draft policy on desysopping somewhere? Has this point been brought up on other CFCs on Wikiversity? [[User:Ottawahitech|Ottawahitech]] ([[User talk:Ottawahitech|discuss]] • [[Special:Contributions/Ottawahitech|contribs]]) 15:45, 16 March 2024 (UTC)
*:: [[User:Ottawahitech]]: Thanks. I do not know of any draft policy on desysopping in the English Wikiversity. Nor did I mention this point elsewhere in Wikiversity; this is a premiere, from what I remember. We had a vote drafted for this in the English Wiktionary, but it did not pass: [[Wikt:en: Wiktionary:Votes/2019-09/Replacing de-sysop votes with confirmation votes]]; but this one passed: [[Wikt:en: Wiktionary:Votes/2020-02/De-sysop votes to pass by simple majority]]; what passed is not so good but better than nothing. My proposal was this: "All votes require a 2/3 supermajority to pass (source), except for de-sysop votes, which shall only require 1/3+1 to pass." --[[User:Dan Polansky|Dan Polansky]] ([[User talk:Dan Polansky|discuss]] • [[Special:Contributions/Dan Polansky|contribs]]) 17:58, 16 March 2024 (UTC)
* {{support|Go for it!}} I have seen you doing useful wikignoming tasks such as categorization and you are also experienced in other wikis. Time to hand superuser to you. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 00:44, 17 March 2024 (UTC)

==== Custodians willing to mentor ====
I am willing to mentor, although with MathXplorer's extensive wiki involvement on Wikiversity and elsewhere, I don't expect much mentoring will be necessary. -- [[User:Jtneill|Jtneill]] - <small>[[User talk:Jtneill|Talk]] - [[Special:Contributions/Jtneill|c]]</small> 03:38, 9 March 2024 (UTC)
: I agree to the mentorship, thank you very much. [[User:MathXplore|MathXplore]] ([[User talk:MathXplore|discuss]] • [[Special:Contributions/MathXplore|contribs]]) 07:02, 9 March 2024 (UTC)

====Outcome====

[[Category:Nominations for Custodianship]]

Bash programming/Function Usage

2024-03-17T00:27:46Z

Elominius: two more functions

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will replace the video files, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:v copy -vn "$2"; }</code>
* Find out the total size of a selection of files: <code>totalsize() { du -sh -c "$@" |tail -n 1; }</code>
* Get the frame rate of a video: <code>ffprobe -v 0 -of csv=p=0 -select_streams v:0 -show_entries stream=r_frame_rate "$1"</code>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2023-12-14T00:58:42Z

Elominius: /* Working with multimedia */ suitable function name

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will replace the video files, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffaudio() { ffmpeg -i "$1" -c:v copy -vn "$2"; }</code>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2023-12-14T00:56:08Z

Elominius: /* Working with multimedia */ interwiki

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:w:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing for privacy. This will replace the video files, so it is recommended to only use it on copies of video files.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -vn "$2"; }</code>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Bash programming/Function Usage

2023-12-14T00:54:44Z

Elominius: /* Examples */ some ffmpeg and multimedia tools

Functions allows you to group pieces of code and reuse it along your programs. <ref>http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-8.html</ref>

== Pre-requisites ==
Now is the time to note, while some concepts are assumed as a pre-requisite, it's reasonable to expect not every student will be familiar with all concepts. Therefore, where a concept is listed here in the pre-requisites, it will be introduced with a link to a more detailed explanation.

'''Environment:'''
* have completed the course Introduction
* use ENVIRONMENT variables
* stdin, stdout
* re-direct outputs
'''Commands:'''
* how to define a function
* '''expr''' -- does arithmetic,
* '''tail''' -- delivers last output lines
* <code>[[history]]</code> -- delivers command history

== New Concepts ==
'''commands:'''
* declare -- to view functions
* history -- to view command history
* set -- positional parameters
* source -- to re-use, or re-load functions
'''shell features'''
* positional parameters
* environment variables
* sub-shell execution

== Use function arguments ==
In this exercise, we will write two functions which use command arguments. The first function will display a function body or bodies. The second will display our command history <ref>https://en.wikipedia.org/wiki/History_(Unix)</ref>.. It will have an optional single argument, a number, which will default to the number of LINES, an [[Wikipedia:Environment variable|Environment variable]], in your terminal window. In the course of the exercise we will:
* use a single function argument
* usa a variable number of arguments
* use a default argument
* use a function to display our function repertoire.

'''View a function body or bodies:'''

First, from our last section, recall:

$ helloWorld () { echo "Hello, World!"; }
$ declare -f helloWorld
... shows the function body,

Does your output meet your expectations. If this is the first time you've tried it, you will see something different than you entered.

Let's write a function to do this:

$ fbdy () { declare -f $*; }
$ fbdy helloWorld fbdy

Now you are less surprised by the output. What's the '''$*''' doing? It says "when this command (or function) is executed, put all the blank-separated positional parameters right here". You observe the function is little different from using the built-in command declare with it's -f option. Like an alias, but the function allows upward-compatible evolution. In this case, a version of '''fbdy''' may return one-line functions as one-liners, another may routinely add (or remove) function tracing to the function body.

Make sure you've executed the examples above.
== Use command history ==
'''How many lines is our terminal displaying?'''

An Environment variable, LINES is often set by the shell to the number of lines the terminal window is displaying.

$ echo $LINES

If you see nothing, then variable is not set on your terminal. We'll supply a default. It is useful to define a function, again almost an alias '''th''', standing for '''T'''ail '''H'''istory. The reason for this function goes a bit beyond our needs here. However it's useful as a short-hand, if not not to provide a consistent interface to different shell's with a different sets of options for a single command.

$ set $(expr ${LINES:-27} - 3); history | tail -$1

execute that command; if needed, look ahead to subshells here is a sample of results:

$ set $(expr ${LINES:-27} - 3); history | tail -$1
683 pushd $(which cmdlib)
684 pushd $(which $(dirname cmdlib))
685 pushd
686 pushd $(dirname $(which cmdlib))
687 view cmdlib
688 ff th
689 history -24
690 th
691 th
692 ff th
693 set | grep HISTSIZE
694 set | grep HIST
695 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
696 declare -f th
697 echo $LINES
698 clear
699 declare -f th
700 th 24
701 th
702 echo $LINES
703 history 24
704 history 45
705 history -h
706 history -T
707 history -x
708 uname -a
709 https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html#index-history-builtins
710 th () { set $1 $(expr ${LINES:-27} - 3; history | tail -$1; }
711 th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
712 echo $LINES
713 set $(expr ${LINES:-27} - 3); history | tail -$1
bin.$

Notice a number of things about that command:
* number 713 is the last command itself
* thirty-one commands were displayed

Now, convert that into a function, and test it.

$ th () { set $1 $(expr ${LINES:-27} - 3); history | tail -$1; }
$ th # returns your recent command history, filling up the screen

It turns out in the bash shell, history is a [[Wikipedia:Shell builtin|builtin]], which shows a complete list of [https://www.gnu.org/software/bash/manual/html_node/Bash-History-Builtins.html bash history builtins here].

What happened here?

The [https://www.mkssoftware.com/docs/man1/set.1.asp set command (ksh version)] in this usage [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html assigns the positional parameters]. If the function is used with one, then its done like this:

$ th 24 # 24 -> $1 ... so, the command becomes "history | tail -24"

If no argument is used, then it becomes

$ th # and if no lines are set, then: expr 27 - 3 ( = 24 ) , so ... "history | tail -24" again

but if LINES was say, 34, then it's [http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html expr] 34 - 3 (= 31) ... history | tail -31

The shell parameter substitution works for named variables (LINES, SHELL, ...) as well as the [https://gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html positional parameters] (1, 2, ... *) and this expression is most useful to [http://wiki.bash-hackers.org/syntax/pe#use_a_default_value assign a default value], in this example:

$ echo ${LINES:-27}

And this last feature introduced here is the ability to insert sub-shell results in the command. The general idea is:

$ command .. $( sub-shell command or commands... ) ...

where the results of ''sub-shell command or commands...'' is inserted into the command ... In our case then, the result of the shell arithmetic is inserted:

$ set "" $(expr $LINES - 3) # $1 was empty, becomes
$ set $(expr 34 - 3) # to be evaluated,
$ set 31; history | tail -$1 # then becomes
$ history | tail -34

== Display functions ==
To collect our new and useful work, lets' see what we have:

$ fbdy fbdy th

For example:

bin.$ fbdy fbdy th
fbdy ()
{
declare -f $*
}
th ()
{
set $1 $(expr ${LINES:-27} - 3);
history | tail -$1
}
bin.$

This is progress. Do your results compare?

== Edit functions ==
== Save functions ==

You have at least two ways to keep a consistent set of functions available for your command line:
* save them in a local file
* save them to load when you login
We'll exercise both methods here. First the local file. You'll find that not all functions are needed in all instances.

=== In a local file ===
Functions can be stored in any <code>.sh</code> file. To load the functions from that file, run the command <code>. filename.sh</code>.

=== load at login ===
Functions can be stored in the file named <code>.bashrc</code>. It is typically located in the home directory (shortcut: <code>~/.bashrc</code>).

== Reload functions ==

== Examples ==
* Create a directory and enter it:
** <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code>
* Change the window title in a terminal emulator:
** <code>window_title() { printf "\033]0;$*\007"; }</code>
* Prevent a hard drive that does not respond to <code>hdparm</code> from spinning down:
** <code>spindisk() { while : ; do (sudo dd if=/dev/$1 of=/dev/null iflag=direct ibs=4096 count=1; sleep 29); done }</code>
** Requires root access to run due to block-level disk access.
** Adjust the time after <code>sleep</code> to just below your hard drive's default spin-down timeout.
** Exit with <kbd>CTRL</kbd>+<kbd>C</kbd>.

=== Working with multimedia ===
* Access the concatenation feature of [[:w:ffmpeg|ffmpeg]] with only two parameters:
** <code>ffconcat() { ffmpeg -f concat -safe 0 -i "$1" -c copy "$2"; }</code>
* Verify the integrity of multimedia files using its decoding mechanism, independently from the file system: <code>fferror() { ffmpeg -v error -i "$1" -f null - ;}</code>
* Generate a table of video resolutions and framerates by processing the text generated by the [[:en:mediainfo|mediainfo]] tool.
** <code>mediainfotable() { mediainfo "$@" |grep -v "Frame rate.*SPF" |grep -P "(name|Width|Height|Frame rate )" |tr '\n' ' ' |sed -r 's/FPS/FPS\n/g' |sed -r "s/( )+//g"; }</code>
* Redact geolocation from video files before sharing. This will replace the video files, so it is recommended to only use it on copy.
** <code>gpsnull() { sed -i -r "s/\+[0-9][0-9]\.[0-9][0-9][0-9][0-9]\+[0-9][0-9][0-9]\.[0-9][0-9][0-9][0-9]\//+00.0000+000.0000\//g" "$@"; }</code>
* Mute the audio of a video (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -an "$2"; }</code>
* Extract the audio from a video file intp a separate file (specify input and output file): <code>ffmute() { ffmpeg -i "$1" -c:v copy -vn "$2"; }</code>

== References ==

{{subpage navbar}}
[[Category:Bash programming]]

Linux/Advice to Windows users

2023-12-03T23:28:22Z

Elominius: possible blackout from connecting or disconnecting an external monitor

Most non-Apple computer users are used to the Mircosoft Windows operating system, since it usually comes pre-included with computers. Should you wish to try out Linux, or install it on a [[:w:live USB|live USB]] (which is difficult with Windows due to licensing restrictions), here are some considerations.

== Distributions ==
There are a variety of distributions of Linux, such as Mint, Ubuntu, and Arch. Due to varying user preferences, it is not possible to have "one big Linux" operating system for everyone.

== Workflow ==
When installing a program on Windows, the usual workflow is to visit a web site through a web browser and download an installer executeable (<code>.exe</code> or <code>.msi</code> file). While a similar concept exists on Linux, ''AppImage'', the typical method of installing software is from ''repositories'' that are accessed by a locally installed ''package manager''. This is usually done through the terminal, but a graphical application manager might be provided or installable.

== Terminal ==
When using Linux, you should familiarize yourself with the terminal (command prompt), since it contains useful and essential commands for searching and managing files, and listing and managing tasks. Examples for common commands include <code>cp</code> for file copying, <code>mv</code> for file moving and renaming, <code>grep</code> for text searching in one or multiple files (more functional than <code>findstr</code> from Windows CMD), <code>find</code> for file searching, <code>wc</code> for word and line counting, <code>ls</code> for file listing, <code>ps</code> for process listing, <code>free</code> for system memory (RAM) statistics, <code>df</code> ("disk free") for total disk usage statistics, and <code>du</code> for individual directories' disk usage.

All these tools are pre-installed as core utilities on Linux, so one can be certain that any computer running Linux has these available.

Like on Windows CMD, the output from terminal commands can be redirected into a text file for later viewing, by appending <code>>></code> and a file name, with or without path, after the command. Additionally, the <code>|tee -a</code> [[:w:Pipeline (Unix)|pipeline]] allows both reading the output in the terminal window and storing it in a text file. Other pipelines are <code>|head</code> for showing the first ten lines only, <code>|tail</code> for the last ten lines, each with the <code>-n</code> parameter for a custom number of lines; <code>|more</code> for pausing after each screen height until a button is pressed, and <code>|less</code> for making the text scrollable using the arrow keys and <kbd>pgUp</kbd> and <kbd>pgDn</kbd>. <code>|wc</code> counts and shows the number of lines, words, and characters in the output to prevent a potentially large output from flooding the terminal window, and can also be used to approximately count files inside a folder (<code>ls -r folder_name |wc</code>).

Multiple pipes can be stacked onto each other. For example <code>|grep</code> can be used multiple times to filter lines by search terms, and then <code>|less</code> to make the results scrollable.

Aliases, variables, and functions are shortcuts which can be specified in the <code>~/.bashrc</code> file which is loaded each time the terminal is opened. "<code>~</code>" is a shortcut for the home directory, usually located at "<code>/home/''username''/</code>". For example, the line <code>alias RLF="readlink -f"</code> creates the command <code>RLF</code> as a shortcut for <code>readlink -f</code>, <code>dl=~/Downloads/</code> creates the shortcut variable <code>$dl</code> for the download folder, and <code>mkcd() { if [ ! -d "$@" ];then mkdir -p "$@" ;fi; cd "$@"; }</code> creates a function command to create and enter a directory in one step. A starting working directory can also be set using the <code>cd</code> ("change directory") command.

Some interactive command-line utilities output and update information directly in the terminal window, such as <code>top</code> and <code>htop</code> for monitoring processes, and <code>iotop</code> for monitoring disk speed usage.

Aliases can also be used to set default parameters or change the behaviour of commands. For example, <code>alias clear="sleep 5;clear"</code> adds a five-second delay to clearing the terminal when using the <code>clear</code> command to allow for interruption when accidentally entered, and <code>cp=cp -n -p -v</code> causes the <code>cp</code> command to be run with the parameters for not overwriting ("no clobber"), for preserving the date and time stamps ("preserve"), and for printing each copied file into the terminal ("verbose").

Command parameters can sometimes be combined. For example, <code>ls -a -l -R</code> has the samee effect as <code>ls -alR</code>.

Using the terminal can be somewhat more difficult at the beginning, but once you are familiar with the commands, getting some tasks done can be faster through the terminal than through graphical user interfaces. For example, once you are somewhat familiar with the parameters of the <code>ffmpeg</code> command-line utility, multimedia conversion can be launched quickly and conveniently through the terminal. So-called [[:w:Wildcard_character|wildcard characters]] such as asterisks ("<code>*</code>") and question marks ("<code>?</code>") allow for the convenient selection of multiple files.

Third-party graphical user interfaces that make use of command-line interface software exist, such as "gparted" for the "parted" command-line utility.

== Proprietary software ==
Some vendor-specific software might not be provided for Linux due to its relatively small user base compared to Windows. Such includes software for controlling the lighting of peripherals like USB keyboards and mice or built-in laptop keyboards, or printer and scanner drivers and control software. Third-party workarounds may or may not exist.

Printers and scanners should support the generic scanning and printing tools pre-included with Linux, but those might not be able to make use of all features of the scanner or printer.

Some professional software such as that by Adobe has limited support for Linux. The "Wine" utility that aims to support Windows software under Linux does not work with all software, so it is not guaranteed to work with a popular proprietary software you might be using. Consider seeking and familiarizing yourself with open-source cross-platform alternatives. Should they lack some functionality, consider filing feature requests at the developers' bug tracker sites.

== Graphics ==
Windows computers typically have the driver for the GPU (graphical processing unit) out of the box, since vendors of Windows computers typically pre-install the graphics driver that matches the graphics card. On Linux, a graphics driver usually has to be installed manually by the user, and video player software and web browsers might have to be configured to make use of it.

Support for very recent graphics cards might be delayed.

Most tasks can be completed without a graphics driver, but playback of video is more power-efficient through the purpose-built graphics card, also known as "hardware decoding", than the more generic CPU (central processing unit), also known as "software decoding". Encoding tasks finish faster and consume less power through hardware encoding.

The CPU (software decoding) might not even be able to play back video at high resolutions such as 4K 2160p smoothly, and the lower efficiency could cause the cooling fans to spin up and is especially detrimental to battery-powered video playback.

Before connecting or disconnecting an untested external monitor for the first time, make sure to save any unsaved information such as text documents beforehand. There might be compatibility issues with some graphics cards, causing your screen(s) to black out and enforcing a reboot, meaning any unsaved information would be lost. If no screen blackout occurs, you can connect and disconnect that monitor safely.

== Dependencies ==
On Windows, [[:w:library (computing)|software libraries]] are typically bundled with downloaded software. On Linux, so-called [[:w:shared object|shared objects]] ("<code>.so</code>" files) are preferred. While this allows for a more efficient memory usage due to less duplication, you might run into what is known as the [[:w:Dependency hell|dependency hell]]. If software developers are uncooperative, they might make their software dependent on different versions of [[:w:Library_(computing)#Shared_libraries|shared libraries]]. Usually, these can be installed simultaneously, but not in all cases. This is known as "conflicting dependencies". Should any dependencies be missing, a package manager might refuse to install any new software. As of 2022, this problem has not been solved yet.

Here is an example of the terminal output when dependency errors occur:
<pre>
The following packages have unmet dependencies:
libbrotli1 : Depends: libc6 (>= 2.29) but 2.23-0ubuntu11.3 is to be installed
libfreetype6 : Depends: libc6 (>= 2.33) but 2.23-0ubuntu11.3 is to be installed
libpng16-16 : Depends: libc6 (>= 2.29) but 2.23-0ubuntu11.3 is to be installed
Depends: zlib1g (>= 1:1.2.11.dfsg) but 1:1.2.8.dfsg-2ubuntu4.1 is to be installed
proftpd-basic : Conflicts: ftp-server
vsftpd : Conflicts: ftp-server
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
</pre>

== References ==
* [https://www.youtube.com/watch?v=s5g5jAB_rdk ''Linux ist KEIN Ersatz für Windows'' – m4xFPS] ("Linux is not a Windows replacement" in German, July 17th, 2022)

[[Category:Linux]]

Backup/Causes of data loss

2023-12-03T19:17:55Z

Elominius: Clarified that cloud can prevent data loss, but the mistake is overrelying on it.

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref name=schneier-201409>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
While storing data both in an online service and locally reduces the risk of data loss compared to storing it only locally, people who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data.<ref name=schneier-201409 />

For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref><ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref><ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref><ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup/Causes of data loss

2023-12-03T18:53:14Z

Elominius: /* Human error */ plural

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computers.<ref>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
People who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data. For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.

<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref>
<ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref>
<ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref>
<ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup/Causes of data loss

2023-12-03T18:52:35Z

Elominius: Even cloud storage is not entirely safe from human error.

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

Human error is also a risk to [[#Overreliance_on_online_services|cloud storage services]], albeit smaller than the risk of data loss on local data storage managed by an individual with little experience in using computer.<ref>{{cite web |url=https://www.schneier.com/blog/archives/2014/09/security_trade-_2.html |title=Security Trade-offs of Cloud Backup |date=2014-09-25 |author=Bruce Schneier |access-date=2023-12-03 }}</ref> For example, Amazon claims their "Glacier" service is "99.99999999999% durable". That is 99 followed by eleven nines, meaning the chance for data loss is claimed to be one in a hundred billion. Even if this claim were true from a technical point of view, which is very unlikely, it does still not take into account that an angry employee could memory-hole data. Though unlikely, it can be safely assumed to be likelier than a hundred billionth.<ref>{{cite web |url=https://www.enterprisestorageforum.com/news/cloud-storage-marketing-hype-amazon-glaciers-big-claims/ |title=Cloud Storage Marketing Hype: Amazon Glacier’s Big Claims |date=2012-09-04 |access-date=2023-12-03 |website=Enterprise Storage Forum |author=Henry Newman }}</ref>

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
People who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data. For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.

<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref>
<ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref>
<ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref>
<ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 |access-date=2023-12-03 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Backup/Causes of data loss

2023-12-03T02:38:36Z

Elominius: + Someone lost their BackBlaze data due to synchronizing with defective RAID storage.

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive. If affordable, RAID network attached storage is for convenient quick access to a large pool of data, but a separate [[:w:cold data|cold storage]] copy that is rarely touched should be kept if the goal is long-term archival.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref><ref name=backblaze20200607 />

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
People who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data. For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.

<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref>
<ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref>
<ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref>
<ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.<ref name=backblaze20200607>{{cite web |url=https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |title= Today I Lost 12 TB of Backblaze "Protected" family Photos |website=Reddit (/r/BackBlaze) |date=2020-06-07 |archive-url=http://archive.today/2022.11.04-062318/https://www.reddit.com/r/backblaze/comments/gykhsd/today_i_lost_12_tb_of_backblaze_protected_family/ |archive-date=2022-11-04 }}</ref>

== Synchronization of bad data ==
A safe backup is independent from its source device. If a backup is automatically synchronized and there is no error detection mechanism, there is a risk of damaged data being synchronized.<ref name=backblaze20200607 />

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Smartphone

2023-11-22T01:07:17Z

Elominius: Avoid covering microphones while filming. Audio quality will suffer.

[[File:Smartphone icon.svg|thumb|right|Smartphone]]

A smartphone is a portable device combining functionality of many solid-state everyday life tools in one unit, including digital camera, camcorder, flashlight torch, music player, book reader, web browser, messenger, radio, environmental sensors like air pressure, and more. The functionality varies among models.

Supplementary to [[:w:Smartphone|the Wikipedia article]], this resource documents tips and guides on usage.

== Charging port care ==
To prevent unnecessary wear on the charging port, connector, and cable endings, grab the handle and not the cable. The two handles are between cable and the metal pieces that enter the port. Push or pull the connector perpendicularly to the charging port without applying sideways pressure that could wear down the connector and port faster.

Slightly and alternatingly moving the connector sideways in parallel to the USB connector's flat surface (longer edge, horizontally to the charging port) is harmless and could facilitate connecting and disconnecting, whereas downward pressure (vertically to charging port against the connector's longer edge) is unhealthy for the connector endings and should be prevented.

Inserting a more worn charging connector wears down the phone's port less, since it requires less insertion force, but has a higher likelihood of undesirably disconnecting.

Wearing down the charging port can be limited through wireless charging. While more convenient, it is slower and produces more heat than wired charging due to the lower efficiency of wireless power transmission. As such, wired charging is more suitable in a hurry.

== Battery care ==
Battery care is of increasing concern as a result of the popularization of designs that deny its replacement, typically the shortest-lived component of the unit.

Signs of an expiring phone battery are quick battery drain and system outages caused by voltage starvation.

In order to delay the battery expiry, stress to it needs to be limited by foregoing fast charging to the extent possible, as it causes heat that stresses the battery, and the operational battery capacity needs to be limited to a centered range between approximately 25% and 75%, as too high and too low charging states wear it down faster. If no automated process is able to limit the charging state, the power needs to be disconnected manually at a charging state of around 70 to 80 percent, as well as be recharged in time to prevent drain below 20%.

The smartphone can optionally be cooled using a portable external USB-powered fan. This both extends the life span of the battery and might increase charging speed if it was throttled by an overheating protection mechanism.

Possible measures to reduce power consumption include deactivating connectivity features such as WiFi and Bluetooth when unneeded, and monitoring background apps' activity. Note that some tasks such as using cellular internet and high-definition video recording inevitably consume much power. Consider bringing an external battery bank ("[[:w:power bank|power bank]]") with you.

When exposed to cold temperatures which could occur during the winter season, a smartphone might power down as a result of the cold temperatures slowing down the chemical reactions in the battery cell, causing an undervoltage condition. Smaller and older batteries are weakened more by cold temperatures. If no external power source is available, it is recommended to warm up the smartphone for a few minutes using your body.

== Physical protection ==
It is recommended to use both a protective case and a screen protector. Even cheap ones provide significant protection compared to none, and are far cheaper and easier to replace than built-in smartphone parts should damage occur.

Smartphone cases may be equipped with useful features such as a kickstand.

When removing a smartphone case that is difficult to remove, apply pressure to either of the shorter shorter edges (i.e. bottom or top side from vertical view), since they are more robust against the pressure.

== Charging speed ==
[[File:ACadapterJUP44100j5createPDCharger20210905TestUSB.jpg|thumb|USB multimeter]]
Using a cable with low electrical impedance (resistance) may speed up charging. Cables rated to support higher currents such as 5 Amp. have thicker copper wires, thus less resistance. Longer cables (e.g. 3 metres) tend to have a higher resistance, though the thickness of the copper wires plays a more important role.

Charging protocols such as [[:w:Charge_controller#Quick_Charge_and_Pump_Express|''Quick Charge'' by Qualcomm and ''Pump Express'' by MediaTek]], which use elevated voltages, are likely compatible with existing USB cables and don't mind long cables, but produce more heat in the end device due to necessary internal voltage downregulation.

Charging speed can be verified using a USB multimeter as depicted.

== Camera ==
=== Controls ===
It has been widely established across smartphone vendors that focusing is done by tapping on an object inside the viewfinder, [[:w:Exposure_compensation|compensating]] the [[:w:exposure value|exposure value]] (brightness) is done by swiping up and down, and focus and exposure are locked by holding on the viewfinder for between half a second to a second.

After tapping on the viewfinder, a rectangle or circle usually denotes the focused position, and it might turn green when focused successfully and otherwise red. The exposure value might remain manually adjustable while focus and exposure are locked. Depending on the camera software, the availability of these gestures could be restricted in video recording mode, slow motion recording mode, and for the front-facing camera.

<ref>{{cite web |title=How to Use Autofocus and Auto Exposure Lock on Your iPhone Camera |url=https://www.iphonelife.com/content/tip-day-why-and-how-use-autofocus-and-auto-exposure-lock-your-iphone-camera |website=www.iphonelife.com |access-date=15 October 2022 |language=en |date=11 August 2015}}</ref>
<ref>{{cite web |title=How to Adjust Exposure on Galaxy S7 • Android Flagship |url=https://androidflagship.com/22675-adjust-exposure-on-galaxy-s7/ |website=Android Flagship |access-date=15 October 2022 |date=6 October 2016}}</ref>

=== Orientation ===
[[File:Example_vertical_video_on_widescreen.JPG|thumb|Appearance of vertical video on wide screen]]
It is recommended to film horizontally and to avoid rotating the unit while recording.

Filming horizontally is recommended as the resulting footage will gracefully fill computer monitors' and televisions' screen surface, whereas vertical video appears as a slit.

Vertically filmed footage would need to be cropped or zoomed in to fill the entire screen, not only resulting in a lower-resolution crop of the video, but depending on video players' capability to do so during playback. Cropping in a video editor either requires time-taking rendering to a separate video, or sufficient processing power for real-time playback.

Orientation should remain constant throughout recording, as filming in a different orientation than when recording was initiated results in a misrotated picture. Counter-measures such as correcting rotation using metadata from the accelerometer or gyroscope have not been implemented as of 2021, and early future implementations might be proprietary (closed-source), thus not widely supported among video players.

To ensure photos and videos are saved in the correct orientation, first look at the user interface's elements. If their orientation does not match the desired orientation, which can happen shortly after launching or when pointing the camera nearly perpendicularly, tilt the device upwards for a short moment to correct the rotation assumed by the software.

Some camera software locks the rotation of the user interface elements while filming or warns the user about rotation.

=== Resolutions ===
The camera software typically lets one adjust the resolution for photography and filming. A higher resolution produces better-quality images and videos, but demands more battery power and storage. Adjust the resolution based on the value of a moment.

If you are low on storage or battery charge, you might want to select a lower resolution to extend the remaining recording time.

; The screen argument
<blockquote>''I don't have a 4K screen, so why should I film in 4K?''</blockquote>

Not having a screen that matches the dimensions of a video resolution option is not an argument not to record in that resolution, since one might acquire such a screen in future, and higher resolutions still deliver a higher quality due to higher [[:w:bit rate|bit rate]]s and [[:w:chroma subsampling|chroma subsampling]]. In addition, higher resolutions allow for more magnification in later [[video editing]], as well as extraction of higher-resolution still pictures. In simple words, if one wants the best possible 1080p (Full HD), one should film in 2160p (4K).

Even if a 2160p video is never watched on a 2160p screen, still frames can be extracted as standalone photographs. 2160p translates into over 8 megapixels, whereas a 1080p frame is just above 2 megapixels.

These benefits make video footage with a higher resolution more ''[[:wikt:future-proof|future-proof]]'', meaning potentially more useful in the future.

The same principle applies to frame rates. While the only purpose of recording at high framerates such as 120 frames per second used to be slowing down the video to create a slow motion effect, screens with a refresh rate of 120 are becoming commonplace on mobile phones, so a video recorded at 120 frames per second in 2015 can be viewed at real-time speed for a smooth effect on a {{#time:Y}} smartphone.

; Purity
[[File:Colorcomp.jpg|thumb|In [{{filepath:Colorcomp.jpg}} full size], this image shows the quality difference between four subsampling schemes.]]
Since the software of smartphone cameras (and other consumer-grade video cameras) encodes the footage with a chroma subsampling scheme of <code>4:2:0</code>, the colour information is stored at only at the quarter of the resolution as the brightness information, in order to compress far more efficiently with only a slight quality loss noticeable to the human eye. This means that 1080p Full HD video (1920×1080 pixels) only has 540p (960×540 pixels) of colour information, whereas 2160p 4K video (3840×2160 pixels) has 1920×1080 pixels of colour information, making it resemble pure <code>4:4:4</code> subsampling at 1080p. Due to the higher bit rate of 2160p 4K video, compression artefacts are less visible even when played back on lower-resolution screens.<ref>[https://www.youtube.com/watch?v=kIf9h2Gkm_U Why Does 4K Look Better on 1080p Monitors – The Daily Mote (May 14th, 2015)]</ref>

=== Microphones ===
Typically, there are two microphone holes, one located at the top edge and one at the bottom edge, to record spatial stereo audio while holding the smartphone horizontally for recording.

Make sure your hands do not cover the microphones of your smartphone while recording video, or the audio quality will suffer.

== Settings ==
Device settings, also referred to as "Preferences", allow improving the convenience of your usage. Settings to consider adjusting, where available, are:

* Enabling the LED indicator during stand-by mode to facilitate finding and picking up the smartphone in dark environments. The LED may blink or stay on continuously. Its power consumption is marginal only, making no noticeable difference in battery endurance.
* Enabling alternative methods for waking up from stand-by, such as double-tap, which takes less effort than pushing the power or home button. Some smartphones may allow waking up through the proximity sensor, which is referred to as "air wake-up" by Samsung, but works while the device is not moving to prevent accidental detection. Smartphones may allow automatically waking up when picked up.
* Specifying applications to be launched with touch gestures detected while the device is in stand-by mode.
* Deactivating transition animations, as they cause inconvenient delays of inertia. These include transitions between menu pages, unfolding the quick control menu, opening applications, and opening the list of recent tasks. On the popular Android operating system and its derivatives, this option is found in the "Developer options", which first have to be revealed by tapping the "build number" entry in the device information menu seven times.
* Deactivating remote volume control by Bluetooth clients such as vehicle audio systems, to be able to retain manual volume control on the smartphone. On Android OS and derivatives, this is also found in the developer options.
* Smartphones may implement gestures recognizable during stand-by mode, also known as "off-screen gestures" or "screen-off gestures". Such may include double-tapping to wake up the device from stand-by mode without having to push any button, and drawing shapes to quickly launch applications such as the camera.<ref>[https://thelitereview.com/phones/ulefone-s7-pro-review-2gb-ram-16gb-rom/ uleFone S7 Pro Review (2GB RAM & 16GB ROM) (2018-06-10)]</ref><ref>[https://www.androidauthority.com/oxygenos-features-879547/ ''OxygenOS: 6 features you need to know about'']</ref>

== Accessories ==
Various accessories could increase convenience and practicality, such as a kickstand case for watching video and typing (depending on angle), a cover, possibly one with preview display for quick assessment of notifications and messages and missed calls, a USB On-the-go adapter for plugging in USB flash drives, and a wireless charging station if supported.

Support for accessories varies among mobile phone models.

== Picking a smartphone ==
If you have difficulty deciding which smartphone to pick, ask yourself which desired functionality is missing from your current smartphone, and what your primary tasks are. If your budget is too low, weigh the importance of features against each other, and get the phone with the more important features.

== Decisions after purchase ==
You may wish to unlock the smartphone's bootloader in order to be able to "root" the device for [[:w:Rooting (Android)#Advantages|administrative tasks such as creating full backups and advanced modifications]]. However, this may void the warranty. It is recommended to make that decision soon after purchase, as switching the bootloader state may require a full erasure of user data, which becomes increasingly difficult to back up with accumulation over time.

== Apps ==
It may be necessary to install multiple apps with a similar purpose, such as file managers, as they functionally may complete each other.

== Screenshots ==
Depending on implementation screenshots may be captured with a short press or an approximately second-long hold of a button combination. The former is preferable, as accidental shots are better than missed shots. Accidental ones can simply be deleted if desirable, whereas missed ones can not simply be recreated.

== Further tips ==
* Do not overrely on a smartphone's water protection. See it as a protection against accidents, but put it into an additional water-protecting case when entering water.<ref>{{Cite web |url=https://www.droidviews.com/is-the-sony-xperia-z2-waterproof/ |first=Adit |last=Sharda |date=2014-05-15 |access-date=2022-01-06 |title=Is the Sony Xperia Z2 Really Waterproof? |language=en-US |website=DroidViews }}</ref>
* Mobile browsers usually allow viewing pages' source code by manually typing <code>view-source:</code> before the URL.<ref>{{cite web |title=How to View Page Source on Mobile browsers? |url=https://techglimpse.com/view-page-source-on-mobile-browsers/ |website=Techglimpse |access-date=6 January 2022 |date=2017-10-11}}</ref>
* Before applying a screen protector, clean it thoroughly using a micro fibre cloth to prevent ugly marks from grains of dust.
* Depending on vendor and configuration, it may be possible to launch the camera quickly to prevent missing moments by pressing the home button or stand-by button twice. Some smartphones feature a dedicated camera shutter button, which may allow for quick launching through double-pressing or holding.<ref>{{cite web |title=Sony Xperia Z2 camera shootout: feature focus |url=https://www.androidauthority.com/sony-xperia-z2-camera-shootout-374656/ |website=Android Authority |access-date=6 January 2022 |date=30 April 2014}}</ref>
* You may choose to photograph and film at lower resolutions to extend the remaining number of photos and recording duration in the remaining data storage space.<ref>{{cite web |title=How Do I Maximize Recording Time On My Recorder? |url=https://www.avertx.com/how-do-i-maximize-recording-time-on-my-recorder/ |website=www.avertx.com |access-date=6 January 2022}} (applies to any camera)</ref>

== See Also ==
* [[Should batteries in portable electronics be replaceable?]]
* [[Wikipedia: Smartphone]]
* [[Wikipedia: List of open-source mobile phones]]

== References ==
{{Reflist}}

[[Category:Smartphones| ]]

Talk:Should batteries in portable electronics be replaceable?

2023-10-14T21:40:53Z

Elominius: empty and forgotten promises

== A more general question. ==

How can we discourage planned obsolescence in consumer goods? The problem extends beyond cell phones. Anecdotally, most consumer goods have quite a limited service life, but even more problematic is the difficulty or impossibility of servicing and repairing such appliances. They are often constructed in a manner that does not make them easy to fix. It would be interesting to discuss how the problem might be addressed with policy, both for the sake of consumer protection and environmental considerations. Ultimately, I think such policy must only apply to larger manufacturers so that it does not create a barrier to entry while still addressing the problem effectively. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]]) 18:44, 26 September 2023 (UTC)

:I don't know how a policy against planned obsolescence is supposed to create a barrier to entry. And if smaller manufacturers were still allowed to create inferior devices with planned obsolescence, who would buy from them rather than established brands?
:Also, what do you consider as the threshold for a "large manufacturer"? Can you make an example? [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 21:24, 26 September 2023 (UTC)

:: It would depend on the policy, but regulation in general tends to do so, for instance, if it involves expensive testing to ensure a product meets certain standards. Smaller businesses should be exempt because otherwise you might end up with an oligopoly as we have in so many industries. "Large" could be defined in many different ways, but I don't see this as a significant problem. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]])

:: In other words, it should be implemented in such a way that it does not allow a small set of large companies to legally enshrine their hold over the market, which they often do because they tend to have a lot of lobbying power and money to spare for kickbacks to grease the crooks in DC. We have weak antitrust policy here in the United States. It's hardly a surprising state of affairs, but we don't have to like it or condone it. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]])

::: Interesting. It could also be that small manufacturers fear accidentally infringing patents they don't even know exist. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 22:47, 26 September 2023 (UTC)

:::: I'm not sure what you mean, but it's perhaps worth remembering that patents only last twenty years before they expire. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]])

:::: On the other hand, a shoestring startup company or some old, small-time domestic manufacturer probably can't afford to send [[wikipedia:Clarence_Thomas#Nondisclosure_of_finances | Thomas Clarence]] or whoever on a vacation to the Bahamas or pay his grand-nephew's tuition money. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]])

Maybe one way to write this sort of policy is to require that certain products come with long warranties that renew if the product must be replaced with a newer model before the warranty is up. This would encourage a reasonably long-lasting product that is also cheaper to repair or refurbish than to replace. I think the EU was considering some laws against planned obsolescence in certain appliances, but I don't know what they consist of. At any rate, I like this idea. [[User:AP295|AP295]] ([[User talk:AP295|discuss]] • [[Special:Contributions/AP295|contribs]])
: Apologies for the late response. There were plans [http://web.archive.org/web/20200227205307/https://www.techradar.com/uk/news/all-new-iphones-might-be-forced-to-have-a-removable-battery three years ago], but those were empty promises that were never acted upon. [[User:Elominius|Elominius]] ([[User talk:Elominius|discuss]] • [[Special:Contributions/Elominius|contribs]]) 21:40, 14 October 2023 (UTC)

Backup/Causes of data loss

2023-10-14T21:34:14Z

Elominius: /* Vulnerable storage location */ another source that proves the point

Causes of data loss include device failure, software bugs, software lock-in, and human error.

== Hardware failure ==
The magnetic head that floats close to a hard disk's metal platters could [[:w:head crash|crash onto it]], the spinning motor's lubrication could wear down, a manufacturing defect could lead to early defect, or a storage controller could malfunction due to a voltage spike caused by a malfunctioning power supply or a lightning strike.

Since RAID storage systems are more complicated, they have more points of vulnerability, such as a malfunctioning controller that could damage data even if the individual drives are working well, or a drive failure poking holes over the entire data stored on the system due to lack of redundancy drives.

This happens because RAID systems typically stripe data across drives to multiply sequential transfer rates, since it allows the individual drives to work in parallel. This performance benefit comes at the cost of vulnerability to data loss, thus requires redundancy drives (or "pairity drives") so the user can swap out a defective drive, and data can be automatically restored on a functional drive.<ref>[https://www.youtube.com/watch?v=gSrnXgAmK8k ''All of our data is GONE!''] – [[:w:Linus Tech Tips|Linus Tech Tips]] – January 4th, 2016</ref>

The complexity of RAID storage systems also makes it more susceptible to maintenance errors.<ref>{{cite web |url=https://louwrentius.com/scrub-your-nas-hard-drives-regularly-if-you-care-about-your-data.html#fnref:raid6 |title= Scrub Your NAS Hard Drives Regularly if You Care About Your Data |date=2020-04-22 |access-date=2023-01-21 |website=Louwrentius.com }}</ref>

== Software bugs ==
Users have reported parts of their mobile phones' internal storage to becoming inaccessible,<ref>[https://forum.xda-developers.com/t/omnia-2-doesnt-recognize-internal-storage.772641/ ''Omnia 2 doesn't recognize internal storage!'' – XDA forum user ''WhiteRussianBC'' – September 2nd, 2010]</ref><ref>[https://forum.xda-developers.com/t/my-memory-8gb-on-ominia-ii-gt-i8000l-has-disappeared.678991/ ''"My Memory" (8GB) on Ominia II (GT-I8000L) has disappeared!'' – XDA forum user ''mattbiondi'' – May 10th, 2010]</ref> folders disappearing during a move,<ref>[https://support.google.com/photos/thread/106201?hl=en ''Photos disappeared after moving to another device folder'' – Gökhun Güneyhan – Google Photos community (April 24th, 2018; 57 replies)]</ref> bogus software updates rendering devices inoperable,<ref>[https://www.youtube.com/watch?v=FeUBdMU2qA0 ''Apple iOS update BRICKS repaired iPhones after screen repair'' - Louis Rossmann (2018-04-10)]</ref> faulty [[:w:colourspace|colourspace]] conversion causing the operating system to crash in perpetuity,<ref>[https://www.youtube.com/watch?v=iXKvwPjCGnY ''How THIS wallpaper kills your phone'' (by Arun Maini, Mrwhosetheboss, 2020-06-04)]</ref> and all contents of their internal storage being deleted.<ref>[https://www.reddit.com/r/Android/comments/16e6kc/i_just_deleted_a_random_folder_in_my_internal/.compact ''I just deleted a random folder in my internal storage and it wiped my internal storage. What the heck just happened?'' – Reddit – /r/Android (January 11th, 2013)]</ref>

Data stored on the memory card is less likely to get lost, since a memory card can be ejected and read externally, as well as being recoverable to some extent using forensic software.

Should a failure such as a system crash or exhausted disk space occur while a file is being written to, the file might be truncated (partially cut off) or blanked. If the file is still opened in the program (for example, text editor), it can be saved on a different device or partition to prevent loss.

Another way to prevent it is creating revisions in separate files by occasionally changing the [[file naming|file name]] while saving. This can be an incremental number or time stamp.

== File system modifications ==
=== Partition management ===
Partition management is a highly risky task.

Before partitions on a data storage device are modified, a full-disk image backup should be taken, since any interruption in the process could destroy the file systems, making the files inaccessible. A full-disk image backup allows for recovery by writing the partitions in their state before the partitioning back to the device.<ref>{{cite web |website=Reddit |date=2022-11-14 |access-date=2023-01-15 |author=islaambaduk_ |title=3 years of data wiped clean*accidentally* |quote=Was partitioning the drive in cmd and the shit happened |url=https://www.reddit.com/r/DataHoarder/comments/yuz10m/3_years_of_data_wiped_cleanaccidentally/.compact }}</ref>

=== File system repair tools ===
Tools such as [[:w:CHKDSK|check disk]] on Windows and [[:w:fsck|file system consistency check]] on Linux intend to repair problems with a file system that could arise after writing to disk was interrupted, which could, for example, lead to inconsistencies between the files and the space marked as used. On Windows, CHKDSK also renames and relocates files with names containing characters Windows considers invalid such as colons and question marks that might have been created on a Linux-based operating system.<ref>[[Wikipedia: CHKDSK]]</ref>

It is recommended to renew the backup before running such tools, since they are not guaranteed to repair a file system successfully and might cause collateral damage. File system repair can be thought of as a digital surgery. Similarly to partition management, an interruption in the process can lead to enormous damage.

However, on [[:w:journaling file system|journaled file systems]] such as NTFS (default on Windows and widely pre-formatted on external drives) and ext4 (default on Linux) and UDF (optical discs), file system checking is usually not needed after a power interruption, and when it is, it is more reliable than on old-fashioned file systems such as FAT32 that lack a journal.

== Human error ==
Human errors that can lead to data loss include the deletion of files and folders that one thinks are already backed up but are not, mistyping a command, choosing a wrong device for writing a disk image to, and dropping a device from a height or into liquid.<ref>[https://www.youtube.com/watch?v=wDYiJRhzXY0 Alles weg - Hochformat (Simon Unge), January 29th, 2020] – smartphone (iPhone XS plus or iPhone 12 plus) stuck on boot screen after falling from a height of several metres onto a carpet. The carpet did cushion the impact, but not enough to prevent damage to the device.</ref><ref>[https://m.youtube.com/watch?v=DVQ1azBUCAI&list=PLkSXFFvjIepOfrbF4fo2y__0xcYGytbYa&index=14&t=2m54s Prank geht schief – BibisBeautyPalace (Bianca Heinicke), December 15th, 2016 (at 2 minutes and 54 seconds)] – smartphone (iPhone 6s) dropped into pool. Afterwards, it powered on but the touch screen was unresponsive to input.</ref><ref>Smartphone fell into lake, resulting in the loss of all data: {{cite AV media |url=https://www.youtube.com/watch?v=pyh1SgMxEZs&t=03m05s |title=Sony Xperia Tablet Z - Eintauchen und Eintauschen |date=2013-05-23 |access-date=2023-06-10 |author=GO2mobile |language=de }} (at 3 minutes and 5 seconds)</ref>

A damaged touch screen of a smartphone after a drop might be insensitive to touch, making it impossible access files. Even when the smartphone is connected to a desktop/laptop computer through a USB cable or wirelessly, the file transfer typically needs to be activated on the phone first, which is impossible if the touch screen is unresponsive.<ref>[https://forum.xda-developers.com/t/q-get-data-from-cracked-nexus-4.2993942/ Get data from cracked Nexus 4 - Panomosh - January 5th, 2015 - XDA developers]</ref>

Before working with sensitive tools such as <code>dd</code> and <code>ddrescue</code>, it is recommended to renew the backup and to unplug any momentarily unneeded external storage devices to protect them from overwriting accidents.

Avoid working with sensitive tools when tired. Fatigue and sleepiness increase the chance of accidents occuring.

On defective devices, data stored on a memory card is likely retrievable externally, therefore storing data on a modular memory card protects against data loss. The most popular memory card format in portable devices is [[:w:MicroSD|MicroSD]].

== Vulnerable storage location ==
Some people make the mistake of treating the internal storage of their computer or mobile phone as a permanent archive.<ref name=davdreamer /><ref name=MatHonan>{{Cite news|url=https://www.wired.com/2012/08/mat-honan-data-recovery/|title=Mat Honan: How I Resurrected My Digital Life After an Epic Hacking|last=Honan|first=Mat|date=2012-08-17|work=Wired|access-date=2023-02-06|language=en-US|issn=1059-1028}} "Had I been regularly backing up the data on my MacBook, I wouldn't have had to worry about losing more than a year’s worth of photos, covering the entire lifespan of my daughter, or documents and e-mails that I had stored in no other location." "And worst of all, my AppleID account was broken into, and my hackers used it to remotely erase all of the data on my iPhone, iPad, and MacBook."</ref><ref>{{Cite web|url=https://discussions.apple.com/docs/DOC-6031|title=Methodology to protect your data. Backups vs. Archives. Long-term data protection - Apple Community|date=2014-06-14|website=discussions.apple.com|access-date=2023-02-06}} "No computer, regardless of HD or SSD size is a data storage device, and should never be considered as such." "All collections of media files such as pictures, music, and videos, unless directly needed should be kept off the notebook and on an external hard drive or likewise."</ref><ref>{{cite web |url=https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |url-status=dead |archive-url=http://web.archive.org/web/20160505014358/https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-music-no-seriously/ |archive-date=2016-05-05 |date=2016-05-04 |title=Apple Stole My Music. No, Seriously. |quote=I painstakingly imported from thousands of CDs and saved to my computer’s internal hard drive. }}</ref>

There is nothing wrong about having redundant copies of files on such devices, and it is useful if accessed frequently. However, without an independent [[:w:cold data|cold data]] backup, files stored on a device's internal storage are at risk of loss from any failure that would make the computer inoperable, ranging from malware, bogus updates, to component failure.

Some vendors of computers and smartphones let the user remotely erase their device if linked from an account, which is intended for use after physical theft. However, this leads to a risk of data being erased through a compromised account.<ref name=MatHonan />

== Bad user interface design ==
Bad user interface design such as the ability to delete photos through a side-swiping gesture on some mobile photo viewers, or the ability to delete history items with a press of the "<kbd>delete</kbd>" key without confirmation on the Firefox web browser could cause the inadvertent deletion of data.

== Software lock-in ==
People might habitually store data inside a locked-in location such as a mobile phone application's internal data, thinking it will always be there. In reality, access to that data could be denied with little warning through a software update that removes a feature, or a software bug.<ref>{{cite web |url=https://www.reddit.com/r/whatsapp/comments/sfi0kp/whatsapp_deleted_all_my_videos_of_a_passed_loved/.compact |title=Whatsapp "deleted" all my videos of a passed loved one. Panicking. |date=2022-01-29 |access-date=2023-01-14 |quote=A person close to me recently passed away and a lot of messages, voice messages, pictures and videos in our conversation were saved on the app and meant a lot to me. Today I noticed I couldn't play videos from our conversation (both the ones I sent and the ones he sent). |author=everyoneelsethatlive |website=Reddit }}</ref>

If data is stored in a locked-in location from where it can not easily be [[:w:data portability|ported]] and is only accessible through specific software, the data will at most only be accessible for the lifespan of the one device it is stored on.

If an app is dependent on an online service, the unavailability or discontinuation of that service could make user data inaccessible, if the app is poorly designed to refuse starting if no connection to the service provider can be established.

Instances of software lock-in include:

* Some mobile browsers like Samsung Internet for Android (earlier known as "S Browser") lock saved pages away in the <code>/data/</code> directory, from where they can not be backed up or copied to other devices, except with root access, which is locked behind the bootloader by default, and unlocking that typically involves a full device erasure.<ref name=sadstate>[https://beepb00p.xyz/sad-infra.html ''The sad state of personal data and infrastructure'' - Karl Icoss, BeepB00p.xyz]</ref>
* In April 2019, software support for the "[[:wikipedia:software as a service|software as a service]]" Telekom Entertain 303 Media Receiver, a digital television apparatus, was deprecated. Since the operating system is software as a service, it stopped working as well, making recordings stored on the hard disk inaccessible. The USB port of the device could not be used for recording or file transfer, and the operating system could also not read USB sticks with existing media. It has no use beyond a 0.5A charger. Data on the device's hard disk was stored in a proprietary format.

The device was introduced in 2011, so people would lose up to eight years of recordings. But since data could not be moved anywhere else from the built-in 500 GB HDD of the device, people were forced to delete recordings anyway to be able to record new footage.<ref>{{cite web |last1=Schamberg |first1=Jörg |title=Telekom zieht 2019 beim alten "Entertain" den Stecker |url=https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |website=onlinekosten.de |access-date=3 October 2022 |archive-url=http://web.archive.org/web/20210412055921/https://www.onlinekosten.de/news/telekom-zieht-2019-beim-alten-entertain-den-stecker_214313.html |language=de-DE |archive-date=12 April 2021 |date=17 January 2019 |url-status=dead}}</ref>

== Overreliance on online services ==
People who have stored data into online services have repeatedly made the mistake of not retaining a local copy, which lead to them losing access to their data. For example, people uploaded [[:w:Home_movies|home videos]] on YouTube without locally retaining the original files. Due to the possibility of videos being removed and accounts being terminated in error, and the difficulty of reaching a YouTube employee for review, a local copy of such videos should be kept.

<ref>[https://www.reddit.com/r/PartneredYoutube/comments/uo9oas/my_youtube_account_was_terminated_for_nudity_or/.compact ''My youtube account was terminated for "nudity or sexual content". Worried about losing thousands of hours of private/unlisted videos for our business.'' – /r/PartneredYouTube – Reddit (May 12th, 2022)]</ref>
<ref>[https://support.google.com/youtube/thread/127327365/my-channel-got-terminated-without-reason-no-strikes-no-warnings-i-worked-4-years-on-this?hl=en ''My channel got terminated without reason, (No strikes, no warnings) I worked 4 years on this .'' (September 23, 2021)] ("''I have old personal videos that I have not saved anywhere else in there saved as private and do not want to lose them.''")</ref>
<ref>[https://twitter.com/Quicksy7/status/1497226199080517635 Tweet from February 25th, 2022]: "''@TeamYoutube can you unban my old youtube account, it has alot of old video memories that i had not saved in my computer yet and all i would really want is to get them back[…]''"</ref>
<ref name=3things>[https://www.youtube.com/watch?v=vDJHHjnWAoc&t=01m02s ''3 Things I Wish I Knew when I First Started on YouTube'' (at 01m02s)] by Tim Schmoyer, Video Creators TV – honeymoon videos taken down due to copyrighted music in background.</ref>

Another example is that Google frequently locks user accounts and demands users to disclose their phone number to regain access. In Germany, it is mandatory since July 2017 to identify oneself to obtain a mobile phone number.<ref>{{cite web |last1=Pieruschka |first1=Von Marius |title=Prepaid-Karten: Ausweispflicht ab 01. Juli 2017 |url=https://www.4g.de/news/prepaid-karten-ausweispflicht-10764/ |website=4G.de – Das offizielle Infoportal zum Thema 4G, LTE, HSPA+ etc. |access-date=3 October 2022 |language=de-DE |date=5 August 2016}}</ref> Other countries have introduced similar laws over time.<ref name=SIM>{{cite web |url=https://www.privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws |title=Timeline of SIM Card Registration Laws |website=Privacy International |date=2019-06-11 |access-date=2023-01-10 |language=en }}</ref> So Google holds accounts hostage until one provides personally identifiable information.<ref>[https://news.ycombinator.com/item?id=17944311 ''Tell HackerNews: Google requiring phone number to log into Chromebook'' – YCombinator – September 9th, 2018]</ref><ref>[https://www.reddit.com/r/privacy/comments/ks7cdt/google_is_forcing_me_to_enter_a_phone_number_to/ ''Google is forcing me to enter a phone number to login to my account'' – Reddit – /r/Privacy – January 7th, 2021]</ref><ref>[https://www.groovypost.com/unplugged/google-stop-asking-phone-number/ ''Google, Stop Asking Me for My Phone Number'' – Austin – GroovyPost – May 14th, 2012] ("''Despite skipping through the process multiple times, Google has continued to harass me with phone number requests. How many times will I have to login and skip it before Google gets the idea? Where is the opt-out button?''")</ref> Internet companies have disrespected users' privacy by using mobile phone numbers for commercial purposes in the past.<ref>{{cite web |url=https://thehackernews.com/2022/05/twitter-fined-150-million-for-misusing.html?m=1 |author=Ravie Lakshmanan |date=2022-05-26 |access-date=2023-01-29 |title=Twitter Fined $150 Million for Misusing Users' Data for Advertising Without Consent }}</ref>

An online platform might change its business model, resulting in a purge of prior data. For example, in 2022, Vimeo planned suspending accounts whose videos consumed excessive bandwidth.<ref>{{cite web |last1=Sato |first1=Mia |title=Vimeo is telling creators to suddenly pay thousands of dollars — or leave the platform |url=https://www.theverge.com/2022/3/15/22979126/vimeo-patreon-creators-price-increase |website=The Verge |access-date=18 June 2023 |date=15 March 2022}}</ref>

Among the largest data loss disasters on Internet social networking services occurred on MySpace. [[:w:Criticism_of_Myspace#Data_loss|The majority of audio tracks uploaded to the site were lost]], reportedly due to a maintenance error. The music tracks were listed but unplayable, with the earliest reports of tracks being unplayable dating back to December 2017.<ref>[https://www.reddit.com/r/Music/comments/7ij6br/myspace_music_profiles/.compact ''MySpace music profiles'' – Reddit post from December 9th, 2017]: "''I have music on a profile from when I was making music […] That is the only copy In Existence and would love to have it again for the memories''"</ref> Initially, the site's developers reported trying to solve the problem, presumably to delay an uproar, but later came to acknowledge that the data was permanently lost.<ref>{{cite web |last1=Brodkin |first1=Jon |title=Myspace apparently lost 12 years' worth of music, and almost no one noticed |url=https://arstechnica.com/information-technology/2019/03/myspace-apparently-lost-12-years-worth-of-music-and-almost-no-one-noticed/ |website=Ars Technica |access-date=24 September 2022 |language=en-us |date=18 March 2019}}</ref> The incident was dubbed a "datapocalypse", a [[:w:portmanteau|portmanteau]] from "data" and "[[:wikt:apocalypse|apocalypse]]".
Even after the web site spontaneously removed the user blogging feature along with existing user blogs without announcement in 2013, nearly five years earlier, some people were still not compelled enough to back up their data locally, resulting in them losing it.<ref>{{cite web |title=MySpace Punishes Its Few Remaining Friends By Vanishing Their Blogs |url=http://techcrunch.com/2013/06/12/bring-the-blogs-back/ |website=TechCrunch |access-date=24 September 2022 |date=12 June 2013}} ([http://archive.today/2014.03.16-194756/http://techcrunch.com/2013/06/12/bring-the-blogs-back/ Livefyre comment archive, 2014])</ref>

Cloud storage providers usually close long unused accounts to clear disk space for new users.<ref>{{cite web |last1=Novet |first1=Jordan |title=Dropbox shows how it manages costs by deleting inactive accounts |url=https://www.cnbc.com/2018/02/23/dropbox-shows-how-it-manages-costs-by-deleting-inactive-accounts.html |website=CNBC |access-date=3 October 2022 |language=en |date=23 February 2018}}</ref> Cloud storage is for short-term convenience, not archival. Exceptions are dedicated paid services with this stated purpose. However, due to lack of control, even those do not replace a local copy.

== Loss of access credentials ==
Data in encrypted archive files or file systems or on a cloud service is at risk of loss if the access credential, commonly a password, is lost.
This is unlikely to happen over a short time span, but significantly likelier in absence in the long term.

As mentioned above, cloud storage providers purge long inactive accounts, meaning by the time one forgets the credentials to a long unused cloud account, it might not exist anymore.

== Loss and theft ==
Portable devices such as mobile phones that store data can be lost or stolen.

This risk is higher during trips, where one might, for example, leave a bag with personal belongings unattended in a crowded area.

The loss of data from such an occurance can be minimized by backing data up to a storage device left at the base (such as a hotel or rented appartment), and synchronized to a cloud storage service if the mobile data plan allows for it.

Although less likely, data could be lost to home burglary as well, especially when stored on the internal storage of expensive devices that are attractive to burglars due to their resale value. Data can only be protected from burglary by storing a copy in a separate location.<ref name=davdreamer>{{cite web |url=https://imgur.com/gallery/edTDPSV |date=2022-07-09 |author=davdreamer |website=Imgur |access-date=2023-01-10 |title=Stolen laptop in Westmeath Ireland, Please Help! |quote=the laptop has five years of photos as well as a book I had just finished writing! }}, "if I could hop in my time machine and back everything up I would."</ref>

== Natural disasters ==
In some locations of the world, people may have to be considerate about natural disasters. For example, a flood disaster could soak a basement, destroying all data storage except for optical discs. This can be prevented using water-tight bags or containers. A basement however is the least vulnerable to an earthquake due to being supported by the surrounding earth. An earthquake mainly targets the top of a house, which could be completely destroyed if the house collapses.

== References ==
<references />

[[Category:Backup]]
[[Category:Information technology]]

Should batteries in portable electronics be replaceable?

2023-10-08T18:22:30Z

Elominius: "power saving mode" argument

{{Wikidebate}}

Batteries in mobile phones, both "non-smart" and [[smartphone]]s, used to be built in a way that would let the user easily replace them.

In 2007, Apple released the first iPhone, and its battery could not be replaced. In the mid-2010s, all major smartphone vendors followed suit by switching from easily replaceable to non-user-replaceable batteries.

Similarly, laptop vendors have switched to difficult-to-replace batteries in the 2010s.

==Batteries should be replaceable==

=== Arguments for ===
* {{Argument for}} - The battery is the shortest-lived part inside portable electronic devices such as smartphones and laptops. No matter how well it is treated, it will lose strength and capacity over time and eventually stop working.
* {{Argument for}} - The battery can be replaced without causing damage to the device. Forcibly opening the sealed back cover of a smartphone with non-replaceable battery causes the seal to be damaged, hence a previously water-resistant smartphone will not be water-resistant anymore.<ref name=S7>{{cite AV media |url=https://www.youtube.com/watch?v=xNPeA4CJ7Ow |title=Can a repaired S7 still be waterproof? |author=Wit Rigs |date=28 September 2016 |access-date=26 September 2023 }}</ref>
* {{Argument for}} - Non-replaceable batteries are a slippery slope. They leave manufacturers possibilities to make batteries even more difficult to replace. Whereas the iPhone 4 from 2010 could be disassembled using a screwdriver to access the battery, the battery in more recent iPhones is only replaceable by breaking a seal, and batteries may be serialized, meaning associated to the smartphone with a serial number so the phone refuses to run or disables some functionality with a replaced battery.<ref>{{cite web |url=https://www.ifixit.com/News/32343/apple-is-locking-batteries-to-iphones-now |title=Apple Is Locking iPhone Batteries to Discourage Repair |author=Craig Lloyd |website=iFixit |date=2019-08-07 |access-date=2023-09-26 }}</ref>
* {{Argument for}} - An empty battery can be replaced with an externally charged battery, so the charge can be renewed within minutes.
** {{Objection}} - Replacing a battery on the go is uncomfortable.
*** {{Objection}} - Walking in public while the smartphone is attached to an external battery pack ("power bank") is uncomfortable as well.
** {{Objection}} - Some smartphones with replaceable battery covers can not be opened with the finger but require a screw to be turned. An example of this is the Samsung Galaxy Xcover series.
*** {{Objection}} - A screwdriver is lightweight to carry and takes less room than a power bank.
*** {{Objection}} - Even with a battery cover that requires screws to open, it takes at most a few minutes to perform a battery replacement.

=== Arguments against ===
* {{Argument against}} - A sealed design is necessary to make a smartphone water-resistant.
** {{Objection}} - Water-resistant smartphones with replaceable batteries such as the Samsung Galaxy S5 and the Sony Xperia V exist.
** {{Objection}} - It is possible to isolate the battery in a way that even if water leaked into the back cover, it could not get to the delicate internal electronics of the phone. Smartphone vendors already did the same with USB charging ports, for example Samsung starting with the Galaxy S5 mini and S7, and Apple with the iPhone 7. Their charging ports need no cover to be water-resistant thanks to being isolated internally.
** {{Objection}} - The water resistance of a sealed smartphone is lost after a battery replacement, since it requires breaking the seal.<ref name=S7 />
** {{Objection}} - Even a sealed design does not guarantee water resistance.<ref>{{Cite web |url=https://www.droidviews.com/is-the-sony-xperia-z2-waterproof/ |first=Adit |last=Sharda |date=2014-05-15 |access-date=2023-09-30 |title=Is the Sony Xperia Z2 Really Waterproof? |language=en-US |website=DroidViews }}</ref> Should water have entered the device, a replaceable battery can be removed to cut off electricity and prevent further damage to the components.
* {{Argument against}} It takes more physical space to build a replaceable battery cover, so a sealed design allows for a larger battery in the same physical space, allowing for thinner smartphones.
** {{Objection}} - The capacity benefit is only temporary due to the battery wearing down.
** {{Objection}} - People purchase smartphones for their functionality, not their thinness. The purpose of a smartphone is to get work done, not to be as thin as possible.<ref>{{cite AV media |url=https://www.youtube.com/watch?v=xmalvGgWIoc |title=Your Smartphone is too thin. Here’s why. |author=Arun Maini |date=2019-08-10 |access-date=2023-09-26 }}</ref>
** {{Objection}} - It could be argued that a vehicle oil tank should not be openable in order to save space. However, oil will expire and need replacement.
* {{Argument against}} One can get the battery replaced at an electronics repair shop.
** {{Objection}} - A phone repair shop could potentially invade the privacy of a client by looking into the user data. This has happened in the past.<ref>{{cite web |url=https://appleinsider.com/articles/21/06/07/apple-pays-millions-to-end-customers-explicit-images-leak-lawsuit |title=Apple pays millions to end customer's explicit images leak lawsuit |author=Malcolm Owne |website=Apple Insider |date=2021-06-07 |access-date=2023-09-26 }}</ref>
** {{Objection}} - Lockdowns during pandemics could cause repair shops to be difficult to access, whereas an easily replaceable battery allows purchasing reserve batteries in advance and storing them at home.
** {{Objection}} - Repair shops are far away from some peoples' location of residence, making them difficult and time-consuming to access.
** {{Objection}} - A battery replacement at an electronics repair store takes far more time, whereas a user-replaceable battery is changed within minutes, sometimes less than a minute, visiting an electronics repair store and waiting for the repair to finish can take hours.
* {{Argument against}} - A user-replaceable battery makes products less economically viable for the vendor. Vendors who make batteries replaceable would earn less money because their users would buy new batteries instead of completely new devices.
** {{Objection}} - An universal outlaw of non-replaceable batteries would "level the playing field" between manufacturers, so no manufacturer would suffer competitive disadvantages.
* {{Argument against}} - The computer or smartphone can be shipped with the battery inserted so it can be booted immediately out of the box.
** {{Objection}} - Inserting the battery on the day the device is unboxed takes less than a minute, which makes no difference in the long term compared to the impending expiry of a battery that can not be replaced without great difficulty.
** {{Objection}} - Nothing prevents a manufacturer from inserting a replaceable battery in advance.
** {{Objection}} - The shipped battery might not be charged.
* {{Argument against}} - Wireless charging can be built right into the phone.<ref>{{cite AV media |title=Samsung Galaxy Unpacked 2015 – Livestream (Replay) |date=2015-03-01 |url=https://www.youtube.com/watch?v=U8mfEud8n4c&t=27m55s |archive-url=https://ghostarchive.org/varchive/youtube/20211117/U8mfEud8n4c |archive-date=2021-11-17 | url-status=live |type=Video |language=en |access-date=2023-09-26 }} (at 27 minutes and 55 seconds) </ref>
** {{objection}} - Nothing prevents a wireless charging battery cover from being shipped within the scope of delivery, and attached to the phone so it has wireless charging out of the box. That wireless charging battery cover can also be made compatible with multiple wireless charging standards.
* {{Argument against}} - A sealed battery is necessary for a premium unibody design with a glass or metal back.
** {{objection}} - Many people apply protective cases to protect their smartphones from damage. The case covers up the "premium" design, making it irrelevant.
** {{objection}} - The materials that are considered "premium" by some people, glass and metal, are more susceptible to damage. Glass can crack after falling to the ground, and metal is vulnerable to bending.<ref>{{cite AV media |url=https://www.youtube.com/watch?v=znK652H6yQM |title=iPhone 6 Plus Bend Test |date=2014-09-23 |access-date=2023-09-26 |author=Lewis George Hilsenteger }}</ref>
** {{objection}} - A "premium" design is irrelevant to the functioning of a smartphone, whereas a functional battery is essential.
** {{objection}} - A "premium" design could distract from productive work.
** {{objection}} - Even to people who don't wear a case on their phones, a glass back ceases to look good as soon as it is by fingerprints, and is slippery, increasing the chance of the phone falling to ground.
** {{objection}} - A metal back interferes with wireless power transmission, making wireless charging impractical and potentially weakening other wireless signals.
* {{argument against}} - A replaceable battery allows a thief to disable anti-theft tracking software by pulling out the battery.
** {{objection}} - Smartphones can typically be force-reset by holding a combination of buttons for a few seconds. This combination of buttons varies among devices.
** {{objection}} - A thief could disable tracking apps by waiting until the battery has drained. The draining of the battery could be accelerated by letting the screen stay on and by opening the camera application.
** {{objection}} - The existence of thieves does not justify making devices difficult to repair. If thieves cause devices to be difficult to repair, thieves have won in the way that they have reduced the life expectancy of devices nearly every person uses every day
* {{argument against}} - People can activate power saving mode to extend the life expectancy of the battery.
** {{objection}} - Power saving mode leads to a deteriorated user experience and slower performance. Why spend money on an expensive smartphone if one is not going to use all of its processing power?
** {{objection}} - Even with power saving mode, the battery will wear down and need replacement, even though later.

== See also ==
* [[Wikipedia: Planned obsolescence]]

== References ==
<references />

[[Category:Electronics]]
[[Category:Battery (electricity)]]

Should batteries in portable electronics be replaceable?

2023-09-30T23:26:54Z

Elominius: clarification; added objection

{{Wikidebate}}

Batteries in mobile phones, both "non-smart" and [[smartphone]]s, used to be built in a way that would let the user easily replace them.

In 2007, Apple released the first iPhone, and its battery could not be replaced. In the mid-2010s, all major smartphone vendors followed suit by switching from easily replaceable to non-user-replaceable batteries.

Similarly, laptop vendors have switched to difficult-to-replace batteries in the 2010s.

==Batteries should be replaceable==

=== Arguments for ===
* {{Argument for}} - The battery is the shortest-lived part inside portable electronic devices such as smartphones and laptops. No matter how well it is treated, it will lose strength and capacity over time and eventually stop working.
* {{Argument for}} - The battery can be replaced without causing damage to the device. Forcibly opening the sealed back cover of a smartphone with non-replaceable battery causes the seal to be damaged, hence a previously water-resistant smartphone will not be water-resistant anymore.<ref name=S7>{{cite AV media |url=https://www.youtube.com/watch?v=xNPeA4CJ7Ow |title=Can a repaired S7 still be waterproof? |author=Wit Rigs |date=28 September 2016 |access-date=26 September 2023 }}</ref>
* {{Argument for}} - Non-replaceable batteries are a slippery slope. They leave manufacturers possibilities to make batteries even more difficult to replace. Whereas the iPhone 4 from 2010 could be disassembled using a screwdriver to access the battery, the battery in more recent iPhones is only replaceable by breaking a seal, and batteries may be serialized, meaning associated to the smartphone with a serial number so the phone refuses to run or disables some functionality with a replaced battery.<ref>{{cite web |url=https://www.ifixit.com/News/32343/apple-is-locking-batteries-to-iphones-now |title=Apple Is Locking iPhone Batteries to Discourage Repair |author=Craig Lloyd |website=iFixit |date=2019-08-07 |access-date=2023-09-26 }}</ref>
* {{Argument for}} - An empty battery can be replaced with an externally charged battery, so the charge can be renewed within minutes.
** {{Objection}} - Replacing a battery on the go is uncomfortable.
*** {{Objection}} - Walking in public while the smartphone is attached to an external battery pack ("power bank") is uncomfortable as well.
** {{Objection}} - Some smartphones with replaceable battery covers can not be opened with the finger but require a screw to be turned. An example of this is the Samsung Galaxy Xcover series.
*** {{Objection}} - A screwdriver is lightweight to carry and takes less room than a power bank.
*** {{Objection}} - Even with a battery cover that requires screws to open, it takes at most a few minutes to perform a battery replacement.

=== Arguments against ===
* {{Argument against}} - A sealed design is necessary to make a smartphone water-resistant.
** {{Objection}} - Water-resistant smartphones with replaceable batteries such as the Samsung Galaxy S5 and the Sony Xperia V exist.
** {{Objection}} - It is possible to isolate the battery in a way that even if water leaked into the back cover, it could not get to the delicate internal electronics of the phone. Smartphone vendors already did the same with USB charging ports, for example Samsung starting with the Galaxy S5 mini and S7, and Apple with the iPhone 7. Their charging ports need no cover to be water-resistant thanks to being isolated internally.
** {{Objection}} - The water resistance of a sealed smartphone is lost after a battery replacement, since it requires breaking the seal.<ref name=S7 />
** {{Objection}} - Even a sealed design does not guarantee water resistance.<ref>{{Cite web |url=https://www.droidviews.com/is-the-sony-xperia-z2-waterproof/ |first=Adit |last=Sharda |date=2014-05-15 |access-date=2023-09-30 |title=Is the Sony Xperia Z2 Really Waterproof? |language=en-US |website=DroidViews }}</ref> Should water have entered the device, a replaceable battery can be removed to cut off electricity and prevent further damage to the components.
* {{Argument against}} It takes more physical space to build a replaceable battery cover, so a sealed design allows for a larger battery in the same physical space, allowing for thinner smartphones.
** {{Objection}} - The capacity benefit is only temporary due to the battery wearing down.
** {{Objection}} - People purchase smartphones for their functionality, not their thinness. The purpose of a smartphone is to get work done, not to be as thin as possible.<ref>{{cite AV media |url=https://www.youtube.com/watch?v=xmalvGgWIoc |title=Your Smartphone is too thin. Here’s why. |author=Arun Maini |date=2019-08-10 |access-date=2023-09-26 }}</ref>
** {{Objection}} - It could be argued that a vehicle oil tank should not be openable in order to save space. However, oil will expire and need replacement.
* {{Argument against}} One can get the battery replaced at an electronics repair shop.
** {{Objection}} - A phone repair shop could potentially invade the privacy of a client by looking into the user data. This has happened in the past.<ref>{{cite web |url=https://appleinsider.com/articles/21/06/07/apple-pays-millions-to-end-customers-explicit-images-leak-lawsuit |title=Apple pays millions to end customer's explicit images leak lawsuit |author=Malcolm Owne |website=Apple Insider |date=2021-06-07 |access-date=2023-09-26 }}</ref>
** {{Objection}} - Lockdowns during pandemics could cause repair shops to be difficult to access, whereas an easily replaceable battery allows purchasing reserve batteries in advance and storing them at home.
** {{Objection}} - Repair shops are far away from some peoples' location of residence, making them difficult and time-consuming to access.
** {{Objection}} - A battery replacement at an electronics repair store takes far more time, whereas a user-replaceable battery is changed within minutes, sometimes less than a minute, visiting an electronics repair store and waiting for the repair to finish can take hours.
* {{Argument against}} - A user-replaceable battery makes products less economically viable for the vendor. Vendors who make batteries replaceable would earn less money because their users would buy new batteries instead of completely new devices.
** {{Objection}} - An universal outlaw of non-replaceable batteries would "level the playing field" between manufacturers, so no manufacturer would suffer competitive disadvantages.
* {{Argument against}} - The computer or smartphone can be shipped with the battery inserted so it can be booted immediately out of the box.
** {{Objection}} - Inserting the battery on the day the device is unboxed takes less than a minute, which makes no difference in the long term compared to the impending expiry of a battery that can not be replaced without great difficulty.
** {{Objection}} - Nothing prevents a manufacturer from inserting a replaceable battery in advance.
** {{Objection}} - The shipped battery might not be charged.
* {{Argument against}} - Wireless charging can be built right into the phone.<ref>{{cite AV media |title=Samsung Galaxy Unpacked 2015 – Livestream (Replay) |date=2015-03-01 |url=https://www.youtube.com/watch?v=U8mfEud8n4c&t=27m55s |archive-url=https://ghostarchive.org/varchive/youtube/20211117/U8mfEud8n4c |archive-date=2021-11-17 | url-status=live |type=Video |language=en |access-date=2023-09-26 }} (at 27 minutes and 55 seconds) </ref>
** {{objection}} - Nothing prevents a wireless charging battery cover from being shipped within the scope of delivery, and attached to the phone so it has wireless charging out of the box. That wireless charging battery cover can also be made compatible with multiple wireless charging standards.
* {{Argument against}} - A sealed battery is necessary for a premium unibody design with a glass or metal back.
** {{objection}} - Many people apply protective cases to protect their smartphones from damage. The case covers up the "premium" design, making it irrelevant.
** {{objection}} - The materials that are considered "premium" by some people, glass and metal, are more susceptible to damage. Glass can crack after falling to the ground, and metal is vulnerable to bending.<ref>{{cite AV media |url=https://www.youtube.com/watch?v=znK652H6yQM |title=iPhone 6 Plus Bend Test |date=2014-09-23 |access-date=2023-09-26 |author=Lewis George Hilsenteger }}</ref>
** {{objection}} - A "premium" design is irrelevant to the functioning of a smartphone, whereas a functional battery is essential.
** {{objection}} - A "premium" design could distract from productive work.
** {{objection}} - Even to people who don't wear a case on their phones, a glass back ceases to look good as soon as it is by fingerprints, and is slippery, increasing the chance of the phone falling to ground.
** {{objection}} - A metal back interferes with wireless power transmission, making wireless charging impractical and potentially weakening other wireless signals.
* {{argument against}} - A replaceable battery allows a thief to disable anti-theft tracking software by pulling out the battery.
** {{objection}} - Smartphones can typically be force-reset by holding a combination of buttons for a few seconds. This combination of buttons varies among devices.
** {{objection}} - A thief could disable tracking apps by waiting until the battery has drained. The draining of the battery could be accelerated by letting the screen stay on and by opening the camera application.
** {{objection}} - The existence of thieves does not justify making devices difficult to repair. If thieves cause devices to be difficult to repair, thieves have won in the way that they have reduced the life expectancy of devices nearly every person uses every day

== See also ==
* [[Wikipedia: Planned obsolescence]]

== References ==
<references />

[[Category:Electronics]]
[[Category:Battery (electricity)]]