{"posts":[{"no":109032656,"closed":1,"now":"06\/11\/26(Thu)16:53:52","name":"Anonymous","sub":"\/asdiq\/ Archiving, storage tech, development, in-depth history\/analysis, and questions general","com":"This is a general which is focused on archiving, but also interested in other related topics.
Storage technology and file sharing:
Hardware, software, services, shadow libraries, backups, home server, and networks such as tape drives, HDDs, file systems, archive.today, IPFS, Arweave, BitTorrent, etc.
Development:
E.g.: web archiving is much harder in 2026 compared to 2016. Too many websites are walled off by systems such as Cl0udflare, making it impossible for services such as archive.is to capture their webpages. That's a big chunk of important data that easily disappears with no web archive captures. We have to develop solutions to this, such as using the SingleFile extension and other stuff.
In-depth history:
E.g.: important web history events and future events such as sites closing, or get into the "minutia and trivia" about the history of websites and all the little changes.
Analysis:
E.g.: analyzing files and folders that you obtained from scraping or data hoarding, or, what you're sad was lost and not archiving, what you're glad was archived.
Questions:
Ask whatever questions about any of this.
Previous
>>108914628<\/a>","filename":"noaudio_crop_scaledown_1772394666845","ext":".mp4","w":572,"h":686,"tn_w":208,"tn_h":249,"tim":1781211232332602,"time":1781211232,"md5":"\/S4XgLc75XJYspoljmbGLA==","fsize":4055258,"resto":0,"archived":1,"bumplimit":0,"archived_on":1781302821,"imagelimit":0,"semantic_url":"asdiq-archiving-storage-tech-development-indepth","replies":29,"images":13},{"no":109032658,"now":"06\/11\/26(Thu)16:54:18","name":"Anonymous","com":"Inspirations for this general:
\/dhg\/ - Data Hoarding General
>Links<\/span>
>Rentry: https:\/\/rentry.org\/dhg<\/span>
><\/span>
>What is \/dhg\/<\/span>
>In this thread we discuss and create technology and software for data-hoarding, archiving, scripts, and more.<\/span>
><\/span>
>gallery-dl - scrape images, manga, videos and more from many websites<\/span>
>https:\/\/github.com\/mikf\/gallery-dl<\/span>
><\/span>
>Hydrus Network<\/span>
>https:\/\/hydrusnetwork.github.io\/hydrus\/<\/span>
><\/span>
>Stash<\/span>
>https:\/\/github.com\/stashapp\/stash<\/span>
><\/span>
>SmartImage<\/span>
>https:\/\/github.com\/Decimation\/SmartImage<\/span>
\/dapp\/ P2P Decentralized Applications General
>Share your favourite dapps here.<\/span>
><\/span>
>Examples:<\/span>
><\/span>
>brig https:\/\/brig.readthedocs.io\/<\/span>
>ipfs https:\/\/ipfs.io\/<\/span>
>ZeroNet https:\/\/zeronet.io\/<\/span>
>Arweave https:\/\/github.com\/ArweaveTeam\/arweave<\/span>
>Gitopia https:\/\/gitopia.org\/<\/span>
>BitTorrent<\/span>
><\/span>
>Leave your suggestions below.<\/span>
><\/span>
>These components collectively make up the future internet known as web3.<\/span>
\/dshag\/ thread
>Data scraping, hoarding and analytics general thread.<\/span>
>What are you scraping, hoarding or analyzing frens? Also post some pics so I can post them from next time, anime also works<\/span>
\/AAD\/ - Archiving And Donating computer resources general
>https:\/\/desuarchive.org\/g\/thread\/108890811\/<\/span>","time":1781211258,"resto":109032656},{"no":109032731,"now":"06\/11\/26(Thu)17:05:50","name":"Anonymous","com":"In the previous thread, I was wonder about a current-day localhost to Internet tunnel which doesn't suck. I wanted to use it for web archiving. I found one!!!
How it ranks:
- Cl0udflare Tunnel (formerly Argo Tunnel): sucks the most, have to put in credit card info for their free tier
- ngrok: sucks less, have to make an account for anything to happen; the free tier has basically unremovable verification walls put up before accessing any of the target webpages
- https:\/\/loca.lt\/ : OK. Same as ngrok except no account is needed
- https:\/\/localhost.run\/ : the best! Only possible problem is subdomains rotate every 5 to 15 minutes, but I don't care about that.","filename":"makesweet-t4vwkn","ext":".gif","w":400,"h":300,"tn_w":125,"tn_h":93,"tim":1781211950905257,"time":1781211950,"md5":"bKLm\/+1UJu0yRsRy2EgMiQ==","fsize":599264,"resto":109032656},{"no":109032787,"now":"06\/11\/26(Thu)17:13:52","name":"Anonymous","com":"Positive:
catbox.moe was recently unexcluded from web.archive.org
Negative:
IA still sucks and is very untrustworthy. 2051 websites are still excluded, and archive.org\/details\/ have done multiple idiotic mass deletions.","filename":"2xg38","ext":".png","w":1276,"h":944,"tn_w":125,"tn_h":92,"tim":1781212432781914,"time":1781212432,"md5":"rnkkdN8Q6tGr7GdO\/hJ\/2Q==","fsize":197467,"resto":109032656},{"no":109033197,"now":"06\/11\/26(Thu)18:25:06","name":"Anonymous","com":">>109032656<\/a>
>video, android<\/span>
Full version of OP video with audio:
https:\/\/web.archive.org\/web\/20260303103254\/https:\/\/chanii.ddns.net\/b\/src\/1772394666845.mp4
It's a cool Cleon scene from some sci-fi TV show:
"How old are you?
I have told you: just over 18,000 years.
You're a liar. Nothing lasts that long."","filename":"jNpBA","ext":".png","w":1024,"h":768,"tn_w":125,"tn_h":93,"tim":1781216706835472,"time":1781216706,"md5":"9XvOHntNsPYWmAOYqtqqeQ==","fsize":28722,"resto":109032656},{"no":109033878,"now":"06\/11\/26(Thu)20:29:41","name":"Anonymous","com":">>109032731<\/a>
One of the commands I run:
$ ssh -R 80:localhost:2016 nokey@localhost.run
I use IPFS and ipwb to make a <s>WARC replay<\/s> webpage capture show up here:
https:\/\/archive.is\/https:\/\/silver-bear-41.gw.ipfs.ninja\/ipfs\/*
https:\/\/web.archive.org\/web\/20260612002528\/https:\/\/80cc6707802495.lhr.life\/memento\/20260612001835\/https:\/\/rule34.xxx\/index.php?page=post&s=view&id=6315729
rule34.xxx is a website which is excluded from web.archive.org, due to artists being pricks or something.","filename":"I5cPV","ext":".png","w":1024,"h":768,"tn_w":125,"tn_h":93,"tim":1781224181434827,"time":1781224181,"md5":"Y5iSDzGJk\/ce87fNCG7rkA==","fsize":14097,"resto":109032656},{"no":109033968,"now":"06\/11\/26(Thu)20:47:20","name":"Anonymous","com":"People do so much bedrotting and doomscrolling: just consuming content from social media and stuff. They lie in bed on their mobile devices, and most people don't archive what they see.
Like a month ago, I found out that the social media platform Threads is more archive-friendly than X (formerly Twitter). Years ago, Twitter was archive-friendly and alt-front-end-friendly (not anymore).
Reminds me, I was talking to some guy IRL and we agreed that:
- Mobile devices like smartphones and tablets = entertainment or amusement devices, locked-down toys which aren't real computers which can do work
- Laptops = a compromise between mobile devices and desktop computers, can do real computational work
- Desktop computers = most productive computers, best type of computer for doing important things
The more portable the computer is, the less powerful it is (both in terms of its own ability and what people can use it for).
>>109032731<\/a>
Version with a screenshot in the locket.","filename":"makesweet-hb75k4","ext":".gif","w":400,"h":300,"tn_w":125,"tn_h":93,"tim":1781225240590974,"time":1781225240,"md5":"nYXHd6Cd70dtR3fsBmrGKw==","fsize":629518,"resto":109032656},{"no":109035503,"now":"06\/12\/26(Fri)02:34:26","name":"Anonymous","com":"Roughly ever week or two, my router automatically resets, restarts, or soft reboots. That results in the ssh transfer speed (and sometimes BitTorrent transfer speed) being significantly faster, or significantly slower. It stays like that until I have another network disconnect + reconnect. Using GNU\/Linux.
I know that router are little computers that people don't really messes with. And like all computers ("smart TV" devices, laptops, not so much mobile devices, etc.) -- if they've been powered on for a long time without restarting -- then they can be glitchy or slow or degraded. That's fixed by a reboot. Then why the change from a fast speed -> router restarts (no Internet connection for like a minute) -> slow speed?
What's up with this?","time":1781246066,"resto":109032656},{"no":109035644,"now":"06\/12\/26(Fri)03:09:48","name":"Anonymous","com":">>109033197<\/a>
>DNA digital data storage<\/span>
Interesting topic.
How much binary data can all the DNA in a human hold?
>https:\/\/m.opnxng.com\/@fromthenext77\/the-astonishing-amount-of-data-in-human-dna-672573c58913<\/span>
>This means that the DNA in a single human cell is roughly equivalent to 750 MB of data.<\/span>
>Now, let\u2019s consider the fact that the human body contains an estimated 37.2 trillion cells<\/span>
>27.9 exabytes (EB)<\/span>
For comparison: Network Storage Power of Filecoin is 14.884 EiB (today, 22.276 EiB in 2025, 23.615 EiB in 2024). 28 EB is twice that.
From what I've read, you need about 1 GB to store all of 1 human's DNA (a single copy of the DNA).
How fast can human biology read a 1-GB DNA thing in bytes per second?
>chatgpt:<\/span>
>DNA replication (S phase): completes genome duplication in around 8 to 10 hours = 26 KB\/s<\/span>
>Per\u2011fork rate: 12.5 bytes\/s per fork (aggregates to said 26 KB\/s)<\/span>
So 25 to 35 kilobytes per second (slow)
Wikipedia:
>In 2021, scientists reported that a custom DNA data writer had been developed that was capable of writing data into DNA at 1 Mbps.<\/span>
That many megabits per second = 125 KB per second (slow or medium speed).
>>109035503<\/a>
Before router reset:
Dealing with 7.38 GiB: 100.00% done in 54m58s
After router reset:
Dealing with 7.19 GiB: 23.20% done, 13h36m45s left
>>109033968<\/a>
A fourth category would be server computers (which are headless and\/or blade). (Server computers are roughly the same thing as desktop computers though.)","filename":"ZuB3yB7QWCk-maxres","ext":".jpg","w":1280,"h":720,"tn_w":125,"tn_h":70,"tim":1781248188472589,"time":1781248188,"md5":"nC3N\/U3XYZv+qSN9XIUHkA==","fsize":107933,"resto":109032656},{"no":109035660,"now":"06\/12\/26(Fri)03:14:29","name":"Anonymous","com":">>109032656<\/a>
bcachefs was the shits but the main developer was solo schizo so it all fell apart.
>native fs encryption<\/span>
>compression<\/span>
>hot\/cold tiers<\/span>
>striping, mirroring, erasure coding<\/span>
>expand or shrink at will<\/span>
>per file\/folder attributes, can say steam folder must stripe data.<\/span>
so much good, so much lost","time":1781248469,"resto":109032656},{"no":109035684,"now":"06\/12\/26(Fri)03:21:11","name":"Anonymous","com":">>109035660<\/a>
I had an "unformatted" 4 TB HDD. Around the time I ran the mkfs.ext4 command I was thinking something like "Should I make this a bcachefs drive, or be lazy and just stick to what I already know: ext4?"
Ended up going with ext4, I kinda regret it but also don't really care. I didn't go with ZFS as it seemed like the type of crappy external hard drive which can't handle ZFS (yes that type of hardware is unfortunately a thing).
(The "lost+found" folder has a timestamp of 2026-06-03 23:15:21 UTC = the time I locked it in as ext4.)","time":1781248871,"resto":109032656},{"no":109035841,"now":"06\/12\/26(Fri)04:04:15","name":"Anonymous","com":">>109032656<\/a>
"The future of data storage is in our DNA | Eitan Yaakobi | TEDxTechnion":
https:\/\/www.youtube.com\/watch?v=gr9-ss4hyPE
https:\/\/dnastorage.technion.ac.il\/
In the video, a theoretical computer scientist talks about how
- We're in the Zettabyte Era [AKA Zettabyte Zone] - https:\/\/en.wikipedia.org\/wiki\/Zettabyte_Era
- "Humanity creates one zettabyte every two days"
- DNA is denser than a hard disk
- 1 gram of DNA can hold 200 petabytes (PB)
- An amount of DNA containing 1 zettabyte = size of a human finger
- DNA will never become obsolete
- DNA printers currently can't make nature's long DNA strands
- Synthetic DNA is lacking error correction or something
- "Nature had the best hard drives all along."
- etc.
>>109035644<\/a>
>DNA digital data storage<\/span>
More on that:
>https:\/\/web.archive.org\/web\/20260519063324\/https:\/\/open.lib.umn.edu\/evolutionbiology\/chapter\/cells-2\/<\/span>
>the estimated number of cells in a blue whale is greater than 100 quadrillion<\/span>
If 750 MB\/cell, then that equals 7.5 x 10^25 bytes = 75 yottabytes (YB).
>>109035644<\/a>
>Filecoin [IPFS-based] is 14.884 EiB today<\/span>
So it's like 10 exabytes less than it was a couple years ago. Filecoin ain't whale to the moon crypto because you can't easily retrieve the data you uploaded: I'm thinking that's one reason for the drop in its Network Storage Power.","filename":"e04178d59527c1cb22accd3b1ebeaab8744f73b1","ext":".jpg","w":320,"h":180,"tn_w":125,"tn_h":70,"tim":1781251455277660,"time":1781251455,"md5":"tXFda7BFyIYXcT3lzRoRog==","fsize":11289,"resto":109032656},{"no":109035860,"now":"06\/12\/26(Fri)04:08:35","name":"Anonymous","com":">>109035841<\/a>
>https:\/\/dnastorage.technion.ac.il\/<\/span>
>Welcome to the DNA Storage Lab! The lab is led by Prof. Eitan Yaakobi and is in the Computer Science Department at the Technion, Taub building.<\/span>
>[...]<\/span>
>The Henry and Marilyn Taub Faculty of Computer Science<\/span>
>Technion - Israel Institute of Technology, Haifa 3200003, Israel<\/span>
>Copyright \u00a9 2022 by Technion DNA Storage Lab. All rights reserved.<\/span>
Jews are planning to harvest goyim's DNA, killing people in the process, and use it to store their data.","time":1781251715,"resto":109032656},{"no":109036083,"now":"06\/12\/26(Fri)05:06:04","name":"Anonymous","com":"Wayback Machine changed its "fake-url" links (direct links to YT video file), now it's
https:\/\/web.archive.org\/web\/2\/http:\/\/vurl.web.archive.org\/youtube\/v\/AsimTPxIza0
>>109035860<\/a>
Or, replace the DNA in some of their goy cattle with whatever data they want to store.","filename":"https---web.archive.org-web-2-http---vurl.web.archive.org-youtube-v-AsimTPxIza0","ext":".png","w":1276,"h":944,"tn_w":125,"tn_h":92,"tim":1781255164549982,"time":1781255164,"md5":"Dj6EBCzzYsR\/u8vXwwgujw==","fsize":115847,"resto":109032656},{"no":109036129,"now":"06\/12\/26(Fri)05:24:28","name":"Anonymous","com":">>108936477<\/a> (cross thread)
In the previous thread, we never found out if TrueCrypt anon brute forced his encrypted file and cracked it by eventually getting the right password.
>>109036083<\/a>
Those do work, for example, this interesting video:
https:\/\/archive.is\/2026.06.12-091352\/https:\/\/web.archive.org\/web\/20240917075121\/http:\/\/vurl.web.archive.org\/youtube\/v\/mSB71jNq-yQ
(The Jew body snatching \/ DNA harvesting posts are jokes in case anyone was dumb enough to think they weren't, or if I was thought to be dumb and serious about it.)
Attached vid is said YouTube video ID AsimTPxIza0 minus audio.","filename":"noaudio_Happiness_only_real_when_shared_idea_of_happiness_Into_the_Wild_Movie_IntoTheWild_shorts_AsimTPxIza0","ext":".webm","w":1080,"h":1440,"tn_w":93,"tn_h":125,"tim":1781256268736513,"time":1781256268,"md5":"7MET6o5mibDOBO1aFARHRQ==","fsize":3504782,"resto":109032656},{"no":109036584,"now":"06\/12\/26(Fri)07:12:22","name":"Anonymous","com":""If you want to build a ship, don't drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea." --Antoine de Saint-Exup\u00e9ry. I wonder about getting more people interested in these topics and activities.
This image is a humorous email from 1995-11-30 15:30 which has a subject line of "FW: Downsizing...A Little Christmas Cheer?"
These photos were deleted off of archive.org\/details\/ by someone other than the uploader (fuck IA).
Original larger photos with metadata in ar:\/\/ (a thing isn't working now, so no TXs):
- 1st page: ...
- 2nd page: ...","filename":"from_bafybeibp3ibajqxwhwam7xx4v6ibsoouqy2zwfuveihrdioypsd4tfo4zm","ext":".jpg","w":2149,"h":1432,"tn_w":125,"tn_h":83,"tim":1781262742785419,"time":1781262742,"md5":"P17wfsIwYSfwKKPeEEjKQw==","fsize":3156966,"resto":109032656},{"no":109036648,"now":"06\/12\/26(Fri)07:29:03","name":"Anonymous","com":"Haven't looked into it a lot, but I think Ghostarchive (general web archive site) at https:\/\/ghostarchive.org\/ uses ReplayWeb.page for webpage captures.
I don't super like ReplayWeb.page as it uses iframes and shadow DOMs (not very portable, interoperate, accessible), but Ghost Archive offers WACZ files to download for each capture, if it operates like I've seen other ReplayWeb.page embeds operate. So that's a positive: can get the source WARC data. It's like if web.archive.org offered WARC files to download for every capture (they don't do that on most captures).","filename":"ErKNh","ext":".png","w":1024,"h":768,"tn_w":125,"tn_h":93,"tim":1781263743959792,"time":1781263743,"md5":"U6KIASfYpi0o+V4k3vCwXw==","fsize":16612,"resto":109032656},{"no":109036766,"now":"06\/12\/26(Fri)07:55:49","name":"Anonymous","com":">>109036648<\/a>
Looking at three random captures:
https:\/\/ghostarchive.org\/archive\/PbspH
https:\/\/ghostarchive.org\/archive\/MDzMs
https:\/\/ghostarchive.org\/archive\/Cn5lo
It seems to use multiple things:
- On two pages: 'Archive system: "Noscript"'
- On one page: 'Archive system: ReplayWeb.Page' = didn't see any WACZ download link
Update: WARC or WACZ download link isn't obvious, but it's there -> https:\/\/ghostarchive.org\/chimurai4\/Cn5lo.warc","time":1781265349,"resto":109032656},{"no":109036849,"now":"06\/12\/26(Fri)08:08:43","name":"Anonymous","com":"Holographic Versatile Disc (HVD)
R.I.P.","filename":"7WBfh","ext":".png","w":1024,"h":768,"tn_w":125,"tn_h":93,"tim":1781266123258379,"time":1781266123,"md5":"bT8EasgFf99c+WpWwPFLeA==","fsize":131950,"resto":109032656},{"no":109036940,"now":"06\/12\/26(Fri)08:22:22","name":"Anonymous","com":">>109036766<\/a>
>seems to use multiple things: ...<\/span>
Oh, I see. Appears that there's two version or views of each capture:
ReplayWeb.Page (same as ...?wr=true):
https:\/\/ghostarchive.org\/archive\/MDzMs
Noscript:
https:\/\/ghostarchive.org\/archive\/MDzMs?wr=false
Neat, so it seems that Ghost Archive has a method of sharing the webpage captures without needing iframes and shadow DOMs (plus a download-able .warc for each capture).","time":1781266942,"resto":109032656},{"no":109036942,"now":"06\/12\/26(Fri)08:22:38","name":"Anonymous","com":">>109035503<\/a>
Point a fan at it. If that doesn't help then replace the PSU.","time":1781266958,"resto":109032656},{"no":109037081,"now":"06\/12\/26(Fri)08:42:52","name":"Anonymous","com":">>109036584<\/a>
Ok this is funny","time":1781268172,"resto":109032656},{"no":109037128,"now":"06\/12\/26(Fri)08:49:29","name":"Anonymous","com":"Cloudflare is fuckin dumb. They will delete your stuff (deplatform you) and extort you. Much of it is a protection racket. Even more proof:
1. Gelbooru: pic related
2. This guy:
>https:\/\/web.archive.org\/web\/20260305171201\/https:\/\/robindev.substack.com\/p\/cloudflare-took-down-our-website<\/span>
>Cloudflare took down our website after trying to force us to pay 120k$ within 24h<\/span>
>TL;DR:<\/span>
>We've been on the Cloudflare Business plan ($250\/month) for years. They suddenly contacted us and asked us to either pay them $120k up front for one year of Enterprise within 24 hours or they would take down all of our domains. While this escalated up our business we had 3 sales calls with them, trying to figure out what was happening and how to reach a reasonable contract in a week. When we told them we were also in talks with Fastly, they suddenly "purged" all our domains, causing huge downtime in our core business, sleepless nights migrating away from CF, irreparable loss in customer trust and weeks of ongoing downtime in our internal systems.<\/span>
3. Stupid thread where people report Gelbooru to Clownflare to try to get it taken down:
>https:\/\/ghostarchive.org\/archive\/CfVMZ?wr=false<\/span>
>https:\/\/web.archive.org\/web\/20260612122804\/https:\/\/ghostarchive.org\/chimurai4\/CfVMZ.warc<\/span>
I found that link from this thread: https:\/\/desuarchive.org\/g\/thread\/107418115\/#107423985
4. See the OP","filename":"1764767762547","ext":".jpg","w":810,"h":1362,"tn_w":74,"tn_h":125,"tim":1781268569095700,"time":1781268569,"md5":"bqqdqf8nf5164fqysVpUPg==","fsize":281416,"resto":109032656},{"no":109037315,"now":"06\/12\/26(Fri)09:20:24","name":"Anonymous","com":">>109036942<\/a>
I'm thinking that overheating isn't the problem. The router is in one of the coolest areas of where I live. I need that previous Internet or LAN speed which was around 14 times faster. Need it for the important data transfers and (archiving) stuff I'm working on.
Around 8 hours after now, I think I'll restart the WiFi thing in my Linux computer (wpa_supplicant). Maybe even unplug the router and plug it back in (last resort as that messes "lots of stuff" up). Feels like the router (or ISP?) is throttling this computer in the LAN as it seems all other computer in the LAN have way faster transfer speeds.","time":1781270424,"resto":109032656},{"no":109037350,"now":"06\/12\/26(Fri)09:26:14","name":"Anonymous","com":">>109037315<\/a>
It's only affecting one PC? Have you looked at network traffic for malware signatures from that machine?","time":1781270774,"resto":109032656},{"no":109037380,"now":"06\/12\/26(Fri)09:32:20","name":"Anonymous","com":">>109037315<\/a>
Disconnect the affected PC from the network and use another machine to ping flood the router.","time":1781271140,"resto":109032656},{"no":109037560,"now":"06\/12\/26(Fri)10:08:23","name":"Anonymous","com":"I asked in \/hsg\/ without any feedback, maybe you guys have some advice for me:
Is there a way to run a 4 disk RAID10 storage solution under Windows with the intention of eventually taking those disks and transferring them into a Synology NAS?
I'm currently using Windows Storage Spaces and would like an exit path out of the windows ecosystem that would allow me to use an array in the meantime.","time":1781273303,"resto":109032656},{"no":109037604,"now":"06\/12\/26(Fri)10:15:24","name":"Anonymous","com":">>109037560<\/a>
How are Synology NAS drives partitioned and formatted? How are your current drives partitioned and formatted?","time":1781273724,"resto":109032656},{"no":109037784,"now":"06\/12\/26(Fri)10:43:48","name":"Anonymous","com":">>109037604<\/a>
My question is more related to interoperability issues between Windows and linux filesystem based raid solutions. I'm sure I could go through all the trouble of setting something up under WSL, but I'd like to know that it's going to transfer to the NAS seamlessly while also not being mismanaged by Windows in the meantime.","time":1781275428,"resto":109032656},{"no":109037800,"now":"06\/12\/26(Fri)10:46:08","name":"Anonymous","com":">>109037784<\/a>
I don't know the answer, but I assume that the Synology NAS will automatically partition and format the drives you install. Test with old drives first if you can't find a definitive answer.","time":1781275568,"resto":109032656}]}