Preservetube - A Youtube archival site.

[danksausage]

you're not really dedicated to the art of archiving

eat a dick

 

[Fields Of Rye]

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

Is there actually any price to cold storage other than the hard drive itself and a place to store them?

If you create a cold storage catalog and put a reasonable price for you to bring it out of said cold storage, all of that shit becomes a shoebox somewhere.

Whats the fail rate of a cold storage device per year? For references, a vital safety component on a nuclear plant has to be .0001% spontaneous failure rate in their life time. How secure do these live streams of DSP need to be?

 

[BigGuyOS 6 : Jovial Janny]

Rather than just a deletions page, have a "quality decrease" page which is much more aggressive about which videos are included, which drastically lowers the quality of the video to take up 5x-10x less space.

If someone is upset about the quality loss, they are free to re-upload the video, which replaces the low quality version with standard quality again and resets the timer on decreasing its quality again for a few months. Or people can email you about things they really care about to get added to an exceptions list which is never reduced in quality.

I second this suggestion. Re-encode to 480p quality if the video has never been watched after a month and it's longer than a specific length. I'd also consider doing content detection and ask if it even needs to be a video archive in the first place. For example, for archives of podcasts or video-essays, the video usually doesn't really matter, but the audio does, so just encode it at 480p and 5FPS. These are the videos taking up most of the space, right?

 

[LongTallTexan]

I discovered PreserveTube independently of KiwiFarms, so I was surprised to see the owner has an account here. One feature I would love to see is the ability to archive in different video formats, for example, 480p. This would both save storage and is usually all that's needed for most videos.

PreserveTube has reached a point of insane unsustainability. It's making nothing, and it's starting to cost more and more. I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

The archive is at 65TB now, and I'm starting to understand why Archive.org stopped accepting YouTube video uploads. People are archiving hundreds and hundreds of videos while watching nothing. There are more videos being archived than watched, and that has to change, or I don't see a reason to keep pumping money into this.

This doesn't mean it's disappearing tomorrow. If it ever comes to that, there'll be a proper month-long notice period, but it's not outside the realm of possibility.

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

Hello @PreserveTube, After checking out your site, I totally understand the problem. You should delete pointless videos like the Quarterings' hundreds of slop videos that consist of reading tweets. The only things that should be saved are old legendary videos like Daft Hands or Take the Blackpill and video essays that randomly got deleted from YouTube, like that one "Queen of 8chan." Besides that, all of the multi-hour Pokémon card opening livestreams can be deleted. I would also suggest capping uploads at 480p.

 

[GizmoG]

Are you running servers yourself or is this all on an S3 Cloud

Amazon's S3 is probably the simplest option to dump low utilization data into. Utilize a bucket class like S3 Glacier Deep Archive to dump any videos that haven't been viewed in at least a year (or some appropriately long enough time for it to be considered dormant). Then, if requested, pull the video from this bucket to wherever you store your normal video content. Throw up some kind of page that notifies the user that the video has been scheduled for retrieval and will be available later. Then present the video as normal once its been fully downloaded and treat it as a regular video.

With pricing like $0.00099 per GB ($0.99 per TB) per month, you could save on monthly costs while long term programmatic solutions are developed (pruning slop, etc). I think having to wait up to 12 hours to pull out a video from storage and have it accessible on the site is a fair compromise (for keeping costs down) if the content hasn't even been looked at in a year. If something becomes relevant again (after years), and such content is requested to be made available, are there really scenarios in which is must be available immediately?

 

[30+GameOvers]

PreserveTube has reached a point of insane unsustainability. It's making nothing, and it's starting to cost more and more. I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

The archive is at 65TB now, and I'm starting to understand why Archive.org stopped accepting YouTube video uploads. People are archiving hundreds and hundreds of videos while watching nothing. There are more videos being archived than watched, and that has to change, or I don't see a reason to keep pumping money into this.

This doesn't mean it's disappearing tomorrow. If it ever comes to that, there'll be a proper month-long notice period, but it's not outside the realm of possibility.

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

I told people PreserveTube is NOT the answer, and to not depend on it. Of course, nobody listened to me and now we see what happens when you put all your eggs into one basket. What happens if the site goes down or the servers shit themselves?
Remember to Archive LOCALLY, keep spare drives. Do not depend on archival sites to do this for you.

 

[Heather Mason 2]

PreserveTube has reached a point of insane unsustainability. It's making nothing, and it's starting to cost more and more. I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

The archive is at 65TB now, and I'm starting to understand why Archive.org stopped accepting YouTube video uploads. People are archiving hundreds and hundreds of videos while watching nothing. There are more videos being archived than watched, and that has to change, or I don't see a reason to keep pumping money into this.

This doesn't mean it's disappearing tomorrow. If it ever comes to that, there'll be a proper month-long notice period, but it's not outside the realm of possibility.

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

I completely respect the idea of preserving literally anything and everything, no matter how trite or inconsequential, but I'm not sure if history will miss 10 hour 1080p streams of Pokemon Red romhacks with zero views. If something has never been watched and is uploaded, or is uploaded and never touched for over a year, maybe it should make way for something more relevant to people actually using the site. Food for thought

Godspeed o7

 

[ono]

One of my favorite videos from like, 2007/2008 is just totally gone from the internet. It was simply titled "Bunny Yelling" iirc, and it's exactly what's on the box; a short video of a woman yelling "Bunnyyyy!" to her pet rabbit, and the rabbit makes some cute surprisingly noisy sounds back. Rabbits don't usually vocalize very much so it was kind of a fascinating video, especially because it was just sitting on the person recording the video. When I tried to search it, instead I found the following:

• Random episodes from TV cartoons that are already available on other piracy sites
• Clips from cartoon TV shows that are, again, already available on other piracy sites
• AMV garbage
• Porn
• Some weird racist videos from Chinese Historian that picked up because he used words like "snow bunny" and "rice bunny"
• Other shitty raceplay videos that were picked up because of keywords like those
• Furry inflation and "weight gain" EPI shit ()
• Vtuber crap
• Other awful gooner content, generally with blurred anime girl tits in the thumbnail
• YouTube-exclusive kids content that nobody is probably going to watch, probably archived from archiver 'tism that it will be gone later
• A few actual rabbit videos, but not the one I was looking for

This was all from just searching the word "Bunny". I searched "Bunny Yelling" first and got nothing. I didn't think to archive the video myself years ago because I never would've guessed that YouTube would take it down. I am pretty sure the video is privated by the owner but I'm not confident. There is a chance they privated it either for personal reasons or from crazies assuming the rabbit was abused somehow to make the sounds. The video is old and very pixelated like 240p or 360p.

The site could probably use some kind of content pruning, particularly for TV shows that are definitely going to be saved somewhere else. The creepy EPI furry videos also deserve to be lost forever.

 

[skunt]

If i were you i'd just delete everything under 20 views. Then change it so that whenever something is archived, it's stored for 3 days and needs 20 votes or else it gets yeeted too. Running a public service is nearly impossible nowadays without some kind of backing because people expect everything on a platter and fuck you if you're not paying for it out of your own pocket.
If you really don't want to delete everything, take all the shit and create a torrent of it and leave it to the community to keep it alive. If they can't keep the torrent(s) alive then it wasn't worth saving in the first place.

 

[i440BX]

Requires lab-like conditions to properly store magnetic tape. Also they don't really last that long. Interesting idea though

Really? I thought just sealing them from dust and magnetizing forces is enough.

 

[Colon capital V]

Maybe look into setting up account creation for archiving videos? Or some way to filter out 'tards from wasting space on the site with 10 hours of Spongebob farting or some shit.

 

[N Space]

2. Youtube transcodes every video people upload for compatibility and storage reasons so if you don't do so already, I think it'd be a good idea. Before uploading clips here, I always transcode videos to AV1 using svt-av1 with a fraction of the original bitrate and virtually the same quality. You can use a cheapo Intel card for hardware transcoding but software encoding compresses better and can be tinkered with. The waiting time for this process may deter people from uploading BS they don't really care about.

3. Idk what your philosophy regarding your site is, but perhaps you could delete music videos with static backgrounds since that's music and not a video IMO or alternatively store them as audio files with an embedded thumbnail to further save space.

To further drive these two points, I made a quick and dirty experiment. I downloaded the following "4K video" which is a Resident Evil track with a neat background. The video file as is takes 101 MiB of memory using the VP9 codec at 1,285 kb/s and ACC audio at 127 kb/s. Transcoding to AV1 with a Constant Rate Factor (CRF) of 60 and OPUS audio at 100 kb/s reduces the memory needed to 16.8 MiB so we're down to 16% of the original size. Going a step further and extracting the background as a PNG image and setting it as a thumbnail gets us to 13.2 MiB or 13% of the original. Finally, since the background image is not really 4K quality we reduce its resolution to 720p and save it as a JPG image getting us down to 7.36 MiB or 7% which is pretty much the memory needed to store the music since it takes 7.18 MiB.

 

[Ted's Cabin]

I second this suggestion. Re-encode to 480p quality if the video has never been watched after a month and it's longer than a specific length.

2. Youtube transcodes every video people upload for compatibility and storage reasons so if you don't do so already, I think it'd be a good idea. Before uploading clips here, I always transcode videos to AV1 using svt-av1 with a fraction of the original bitrate and virtually the same quality.

As already suggested, dump the quality down to 480, or even 360.

If I can make a suggestion, maybe allow a selectable quality selection when preserving, or possibly downgrade certain videos? I'd rather have 480 p video's of plenty of stuff than have everything be 1080 but a storage crisis.

Preservetube already downloads low quality 480p or 360p VP9 streams from Youtube automatically. The fact that no one here knows this kind of proves that to date not a single person has watched a video on PreserveTube.

 

[Sooty Soot]

PreserveTube has reached a point of insane unsustainability. It's making nothing, and it's starting to cost more and more. I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

The archive is at 65TB now, and I'm starting to understand why Archive.org stopped accepting YouTube video uploads. People are archiving hundreds and hundreds of videos while watching nothing. There are more videos being archived than watched, and that has to change, or I don't see a reason to keep pumping money into this.

This doesn't mean it's disappearing tomorrow. If it ever comes to that, there'll be a proper month-long notice period, but it's not outside the realm of possibility.

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

Did the number of archives go down when you added that warning/popup that shows up when you go to archive a video? I know that it stopped me from archiving normally low value videos that I would've archived for the sake of "Archive everything". Personally, I wouldn't mind if you put up banner ads on the site if it means we can still keep archiving videos.

 

[p1138]

Preservetube already downloads low quality 480p or 360p VP9 streams from Youtube automatically. The fact that no one here knows this kind of proves that to date not a single person has watched a video on PreserveTube.

I guess? But at the same time there's no reason to unless the original got deleted.

 

[Lemmingwiser]

I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

Isn't that the inherent nature of it? I never open files in my backup hard disk that I keep at a different location than my home either. But when the house burns down, it'll be the only copy. Isn't that kinda the point of it?

Wouldn't costs skyrocket if people were engaging with them a lot on top of it?

 

[Spungus]

Amazon's S3 is probably the simplest option to dump low utilization data into. Utilize a bucket class like S3 Glacier Deep Archive to dump any videos that haven't been viewed in at least a year (or some appropriately long enough time for it to be considered dormant). Then, if requested, pull the video from this bucket to wherever you store your normal video content. Throw up some kind of page that notifies the user that the video has been scheduled for retrieval and will be available later. Then present the video as normal once its been fully downloaded and treat it as a regular video.

The issue with S3 Glacier is that they get you hard on ingress/egress, costing $20 to retrieve a TB if the videos are on demand (relatively). You could offset these costs by adding them to a queue and once you have say, 1TB of requests in bulk, they only charge around $2.50/TB for retrieval, but still, that's difficult to work with.

Though as I think about it more, it might work, assuming people are willing to contribute. Make it obvious the goal is long term archiving, not a proxy, and that unviewed videos will go into the archive and cost money to retrieve. Let people buy credits, and give a "I want this eventually" vs "I want this ASAP", the former bundles requests together into 1TB bulk requests and then emails the people when it's live for a lower price.

 

[Ted's Cabin]

I guess? But at the same time there's no reason to unless the original got deleted.

I definitely agree. It's just strange that the guy running an archive service had to learn that most material is not accessed until it's lost everywhere else and someone is looking for it.

Unless it's for some internet history, PreserveTube is incompatible with ARCHIVE FUCKING EVERYTHING

You can't actually archive everything without basically having the same billion dollar infrastructure as Youtube. A very small subset has to be selected and that means you have to have some system to determine what's actually at risk or worth keeping.

Tossing your dollars at an unstable site whose owner can pull the trigger anytime is a pretty funny thing to do.

Did you forget where you're posting? This forum only exists because of people willing to throw money at a supreme retard who can shut everything down at any time.

You're correct. Never trust internet archival unless it's on your own hard drive.

It's also not "internet archival" if it only exists on some guy's hard drive. It has to be publicly available at some point. I have tons of data that will only ever be useful if every other online copy is destroyed, and even then it only becomes useful once it's actually reuploaded somewhere.

 

[Mark Goldbridge]

PreserveTube has reached a point of insane unsustainability. It's making nothing, and it's starting to cost more and more. I wouldn't mind paying for it if the videos were actually being watched, but there are tens of TBs of videos that have never been watched, ever.

The archive is at 65TB now, and I'm starting to understand why Archive.org stopped accepting YouTube video uploads. People are archiving hundreds and hundreds of videos while watching nothing. There are more videos being archived than watched, and that has to change, or I don't see a reason to keep pumping money into this.

This doesn't mean it's disappearing tomorrow. If it ever comes to that, there'll be a proper month-long notice period, but it's not outside the realm of possibility.

If anyone has any ideas on how this can be worked on, I'd be more than open to them.

Block all IPs from majority non-White countries, trust me they never pay a dime.

 

[Enne]

I think some metrics to determine on whether or not to keep a video is probably the best idea. If you're going to shut it down you might as well attempt to start pruning videos and see if it's more sustainable/still useful.

Also potentially re-encoding videos may help (if this hasn't been done already), also maybe make an account system that just tracks what you've uploaded or any videos you want to keep track of then if anything is going to get pruned then show it on the site and let people do a captcha or something to keep it. Alternatively, pay to make an archival permanent or extend its lifetime before being pruned. I feel like there's a few things you could try (I would avoid email notifications for pruning though because that costs money).