Archive.is blog
-
Can your spider be stopped in a similar way as IA Archiver: "User-agent: ia_archiver Disallow: /" Thank you.
Anonymous
There is no spider (as a machine which takes decisions what to archive).
All the urls are entered manually by users (or taken from https://en.wikipedia.org/wiki/Special:RecentChanges, where they also appear as a result of user edits).
If the archive would check and obey robots.txt, then, if archiving is disallowed, the error should be seen to the user, right?
Then, on seeing the error, the user will archive the page indirectly, first feeding the url to an url shortener (bit.ly, …) or to an anonimizer (hidemyass.com, ..) or to another on-demand archive (peeep.us, …), and then archive the same content from another url, thus bypassing the robots.txt restrictions.
So, this check will not work the same way as it works with IA Archiver (which is actually a machine which takes decisions).
-
Where may I download the API/Source code to this site? Anonymous
It is not open source. May be I will release the current version as open source after switching to a newer one.
About API: you can submit a page by issuing a HTTP request with any suitable tool like curl or wget: http://blog.archive.is/post/45031162768/can-you-recommend-the-best-method-script-so-i-may-batch
It you need a more complex API, to perform searches, get answers in xml or json, etc, just let me know, I will implement it or find a way how to it.
-
Congrats for keeping this as a free service, this a very useful site. You said in your blog that this cost you $150 to $300 a month with more than 10TB of saved data. I'm just curious how do you manage to get this, considering that any reliable cloud hosting is around $0.1 per GB per month ? Anonymous
cloud hostings are overpriced compared to prices of dedicated servers and to collocation own hardware(an extra 3Tb disk costs about $200 and it will serve you a few years)
-
It seems that Twitter feeds with a lot of tweets (500+) are not fully archiving, archival attempts seem to get stuck on a page with the loading spinner still displaying. Can this get fixed by increasing a time-out value or something? Anonymous
This is one of the problems I plan to fix by updating the web browser core (may be in a week or two).
The one which is currently used is quite old and has obsolete and slow JavaScript interpreter. Long Twitter pages causes consuming of gigabytes of RAM and 100% CPU. I do not think increasing of timeouts would help - there is a risk of the browser gets stuck forever or crashes because of out of memory.
-
Your service is awesome! Is it possible to donate? The one thing I feel is missing is a feature for archiving pdfs. Perhaps it's a manual or a terms of service I'd really like it if there was an independent copy that could back me up. Anonymous
Hi.
You can use webcitation.org to store pdf.
-
Pages from reddit.com will be saved with all comments expanded
Before the fix: http://archive.is/0gmIN
After the fix: http://archive.is/eRCSv
-
what happened to the site? Anonymous
it works for me (and for many others as well)
-
Added “download .zip” function
-
You have typo in FAQ: “It there any limit on the page size ?” should be “Is there (…)”. kthx. Anonymous
Thank you
-
what is the name of the archive is robot? Anonymous
It used to be “archive.is bot”.
Now it impersonates as a regular browser, because some popular sites (instagram to name one) try to detect if the page was requested by bot or human and in the former case can show the ugly version of page which is optimized for bots, not for people.
