Caching Proxy

The most ambitious & total approach to local caching is to set up a proxy to do your browsing through, and record literally all your web traffic; for example, using Live Archiving Proxy (LAP) or WarcProxy which will save as WARC files every page you visit through it. (Zachary Vance explains how to set up a local HTTPS certificate to MITM your HTTPS browsing as well.)

One may be reluctant to go this far, and prefer something lighter-weight, such as periodically extracting a list of visited URLs from one’s web browser and then attempting to archive them.

Batch job downloads

For a while, I used a shell script named, imaginatively enough, local-archiver:

#!/bin/sh
set -euo pipefail

cp `find ~/.mozilla/ -name "places.sqlite"` ~/
sqlite3 places.sqlite "SELECT url FROM moz_places, moz_historyvisits \
                       WHERE moz_places.id = moz_historyvisits.place_id \
                             and visit_date > strftime('%s','now','-1.5 month')*1000000 ORDER by \
                       visit_date;" | filter-urls >> ~/.tmp
rm ~/places.sqlite
split -l500 ~/.tmp ~/.tmp-urls
rm ~/.tmp

cd ~/www/
for file in ~/.tmp-urls*;
 do (wget --unlink --continue --page-requisites --timestamping --input-file $file && rm $file &);
done

find ~/www -size +4M -delete

The code is not the prettiest, but it’s fairly straightforward:

• the script grabs my Firefox browsing history by extracting it from the history SQL database file¹², and feeds the URLs into wget.

  wget is not the best tool for archiving as it will not run JavaScript or Flash or download videos etc. It will download included JS files but the JS will be obsolete when run in the future and any dynamic content will be long gone. To do better would require a headless browser like PhantomJS which saves to MHT/MHTML, but PhantomJS refuses to support it and I’m not aware of an existing package to do this. In practice, static content is what is most important to archive, most JS is of highly questionable value in the first place, and any important YouTube videos can be archived manually with youtube-dl, so wget’s limitations haven’t been so bad.
• The script splits the long list of URLs into a bunch of files and runs that many wgets because wget apparently has no way of simultaneously downloading from multiple domains.

• The filter-urls command is another shell script, which removes URLs I don’t want archived. This script is a hack which looks like this:

  #!/bin/sh
  set -euo pipefail
  cat /dev/stdin | sed -e "s/#.*//" | sed -e "s/&sid=.*$//" | sed -e "s/\/$//" | grep -v -e 4chan -e reddit ...

A local copy is not the best resource - what if a link goes dead in a way your tool cannot detect so you don’t know to put up your copy somewhere? But it solves the problem pretty decisively.

Daemon

archiver has an extra feature where any third argument is treated as an arbitrary sh command to run after each URL is archived, to which is appended said URL. You might use this feature if you wanted to load each URL into Firefox, or append them to a log file, or simply download or archive the URL in some other way.

For example, instead of a big local-archiver run, I have archiver run wget on each individual URL: screen -d -m -S "archiver" sh -c 'while true; do archiver ~/.urls.txt gwern@gwern.net "cd ~/www && wget --unlink --continue --page-requisites --timestamping -e robots=off --reject .iso,.exe,.gz,.xz,.rar,.7z,.tar,.bin,.zip,.jar,.flv,.mp4,.avi,.webm --user-agent='Firefox/3.6' 120"; done'. (For private URLs which require logins, wget can still grab them with some help: installing the Firefox extension Export Cookies, logging into the site in Firefox like usual, exporting one’s cookies.txt, and adding the option --load-cookies cookies.txt to give it access to the cookies.)

Alternately, you might use curl or a specialized archive downloader like the Internet Archive’s crawler Heritrix.

Cryptographic timestamping local archives

We may want cryptographic timestamping to prove that we created a file or archive at a particular date and have not since altered it. Using a timestamping service’s API, I’ve written 2 shell scripts which implement downloading (wget-archive) and timestamping strings or files (timestamp). With these scripts, extending the archive bot is as simple as changing the shell command:

@reboot sleep 4m && screen -d -m -S "archiver" sh -c 'while true; do archiver ~/.urls.txt gwern2@gwern.net \
        "wget-archive" 200; done'

Now every URL we download is automatically cryptographically timestamped with ~1-day resolution for free.

Resource consumption

The space consumed by such a backup is not that bad; only 30-50 gigabytes for a year of browsing, and less depending on how hard you prune the downloads. (More, of course, if you use linkchecker to archive entire sites and not just the pages you visit.) Storing this is quite viable in the long term; while page sizes have increased 7x between 2003 and 2011 and pages average around 400kb¹³, Kryder’s law has also been operating and has increased disk capacity by ~128x - in 2011, $80 will buy you at least 2 terabytes, that works out to 4 cents a gigabyte or 80 cents for the low estimate for downloads; that is much better than the $25 annual fee that somewhere like Pinboard charges. Of course, you need to back this up yourself. We’re relatively fortunate here - most Internet documents are born digital and easy to migrate to new formats or inspect in the future. We can download them and worry about how to view them only when we need a particular document, and Web browser backwards-compatibility already stretches back to files written in the early 1990s. (Of course, we’re probably screwed if we discover the content we wanted was dynamically presented only in Adobe Flash or as an inaccessible cloud service.) In contrast, if we were trying to preserve programs or software libraries instead, we would face a much more formidable task in keeping a working ladder of binary-compatible virtual machines or interpreters¹⁴. The situation with digital movie preservation hardly bears thinking on.

There are ways to cut down on the size; if you tar it all up and run 7-Zip with maximum compression options, you could probably compact it to 1/5th the size. I found that the uncompressed files could be reduced by around 10% by using fdupes to look for duplicate files and turning the duplicates into a space-saving hard link to the original with a command like fdupes --recurse --hardlink ~/www/. (Apparently there are a lot of bit-identical JavaScript (eg. JQuery) and images out there.)

Good filtering of URL sources can help reduce URL archiving count by a large amount. Examining my manual backups of Firefox browsing history, over the 1153 days from 2014-02-25 to 2017-04-22, I visited 2,370,111 URLs or 2055 URLs per day; after passing through my filtering script, that leaves 171,446 URLs, which after de-duplication yields 39,523 URLs or ~34 unique URLs per day or 12,520 unique URLs per year to archive.

This shrunk my archive by 9GB from 65GB to 56GB, although at the cost of some archiving fidelity by removing many filetypes like CSS or JavaScript or GIF images. As of 22 April 2017, after ~6 years of archiving, between xz compression (at the cost of easy searchability), aggressive filtering, occasional manual deletion of overly bulky domains I feel are probably adequately covered in the IA etc, my full WWW archives weigh 55GB.

URL sources

#!/bin/sh
set -euo pipefail

cp --force `find ~/.mozilla/firefox/ -name "places.sqlite"|sort|head -1` ~/
sqlite3 -batch places.sqlite "SELECT url FROM moz_places, moz_historyvisits \
                       WHERE moz_places.id = moz_historyvisits.place_id and \
                       visit_date > strftime('%s','now','-1 day')*1000000 ORDER by \
                       visit_date;" | filter-urls
rm ~/places.sqlite

@hourly firefox-urls >> ~/.urls.txt

linkchecker --check-extern -odot --complete -v --ignore-url=^mailto --no-warnings http://www.longbets.org
    | fgrep http # [
    | fgrep -v -e "label=" -e "->" -e '" [' -e '" ]' -e "/ "
    | sed -e "s/href=\"//" -e "s/\",//" -e "s/ //"
    | filter-urls
    | sort --unique >> ~/.urls.txt

Locality

Spatial locality can be subtle. For example, natural language text, though not structured line-by-line as visibly as a dictionary, is still far from random and has many local correlations a compression tool might be able to pick up. If this is true, we would expect that with a sufficiently smart compressor, a text file would compress better in its natural form than if it were randomly shuffled. Is this the case, or are compression utilities are too stupid to see any different between random lines and English prose? Taking 14M of text from gwern.net, we can see for ourselves:

# uncompressed
$ cat *.page */*.page */*/*.page |                                                 wc --bytes
13588814

# compressed, files in lexicographic order
$ cat *.page */*.page */*/*.page |                      xz -9 --extreme --stdout | wc --bytes
3626732
# compressed, all lines sorted alphabetically
$ cat *.page */*.page */*/*.page | sort               | xz -9 --extreme --stdout | wc --bytes
3743756
# compressed, all lines randomly shuffled except for non-unique lines sorted together
$ cat *.page */*.page */*/*.page | sort --random-sort | xz -9 --extreme --stdout | wc --bytes
3831740
# compressed, all lines randomly shuffled
$ cat *.page */*.page */*/*.page | shuf               | xz -9 --extreme --stdout | wc --bytes
3862632

The unmolested text compresses to 3.626M, but the same text randomly shuffled is 3.862M! I also included an intermediate form of randomization: despite the name, the --random-sort option to sort is not actually a random shuffle but a random hash (this is not documented in the man page, though it is in the info page) and so any repeated non-unique lines will be sorted together (allowing for some easy duplication deletion), and so we would expect the only-partial randomization of --random-sort to maybe perform a bit better than the true random shuffle of shuf. And it does.

Web archives

Spatial locality also applies to our web archiving. If you are mirroring websites, or otherwise compiling a lot of directories with redundant data on a file-by-file level, there’s a cute trick to massively improve your compression ratios: don’t sort the usual lexicographic way, but sort by a subdirectory. (I learned about this trick a few years ago while messing around with archiving my home directory using find and tar.) This is one of the issues with archiving gigabytes of crawls from thousands of domains: URLs have a historical oddity where they are not consistently hierarchical. URLs were originally modeled after hierarchical Unix filesystems; this page, for example, lives at the name /home/gwern/wiki/Archiving-URLs.page, which follows a logical left-to-right pattern of increasing narrowness. If one lists my entire filesystem in lexicographic order, all the files in /home/gwern/ will be consecutive, and the files in wiki/ will be consecutive, and so on. unfortunately, the top level of URLs breaks this scheme - one does not visit https://com/google/mail/?shva=1#search/l%3aunread, one visits https://mail.google.com/mail/?shva=1#search/l%3aunread; one does not visit http://net/www/gwern/Archiving-URLs but https://www.gwern.net/Archiving-URLs. So if I download a.google.com and then later z.google.com, a lexicographic list of downloaded files will separate the files as much as possible (even though they are semantically probably similar). A quick example from my current WWW archive:

$ ls
...
typemoon.wikia.com/
tytempletonart.wordpress.com/
ubc-emotionlab.ca/
ubook.info/
ucblibrary3.berkeley.edu/
uhaweb.hartford.edu/
ujsportal.pacourts.us/
ukpmc.ac.uk/
uk.reuters.com/
ultan.org.uk/
...

The directories are indeed sorted, but aside from the initial 2 letters or so, look nothing like each other: a Wikia subdomain rubs shoulders with a WordPress blog, a .ca domain is between a .com, a .info, and a .edu (with a .us and .uk thrown in for variety), and so on. Is there any way to sort these directories with a bit of parsing thrown in? For example, maybe we could reverse each line? Some web browsers store URLs reversed right-to-left to enable more efficient database operations, as do Google’s BigTable systems¹⁵ (to assist their relatively weak compression utility Snappy). Turns out GNU sort already supports something similar, the --key & --field-separator options; the man page is not very helpful but the info page tells us:

'-t SEPARATOR'
'--field-separator=SEPARATOR'
     Use character SEPARATOR as the field separator when finding the
     sort keys in each line.  By default, fields are separated by the
     empty string between a non-blank character and a blank character.
     By default a blank is a space or a tab, but the 'LC_CTYPE' locale
     can change this.

     That is, given the input line ' foo bar', 'sort' breaks it into
     fields ' foo' and ' bar'.  The field separator is not considered
     to be part of either the field preceding or the field following,
     so with 'sort -t " "' the same input line has three fields: an
     empty field, 'foo', and 'bar'.  However, fields that extend to the
     end of the line, as '-k 2', or fields consisting of a range, as
     '-k 2,3', retain the field separators present between the
     endpoints of the range.


'-k POS1[,POS2]'
'--key=POS1[,POS2]'
     Specify a sort field that consists of the part of the line between
     POS1 and POS2 (or the end of the line, if POS2 is omitted),
     _inclusive_.

     Each POS has the form 'F[.C][OPTS]', where F is the number of the
     field to use, and C is the number of the first character from the
     beginning of the field.  Fields and character positions are
     numbered starting with 1; a character position of zero in POS2
     indicates the field's last character.  If '.C' is omitted from
     POS1, it defaults to 1 (the beginning of the field); if omitted
     from POS2, it defaults to 0 (the end of the field).  OPTS are
     ordering options, allowing individual keys to be sorted according
     to different rules; see below for details.  Keys can span multiple
     fields.

     Example:  To sort on the second field, use '--key=2,2' ('-k 2,2').
     See below for more notes on keys and more examples.  See also the
     '--debug' option to help determine the part of the line being used
     in the sort.

Hence, we can do better by ordering sort to break on the dots and focus on the second part of a URL, like so:

$ ls | sort --key=2 --field-separator="."
...
uhaweb.hartford.edu/
adsabs.harvard.edu/
chandra.harvard.edu/
cmt.harvard.edu/
dash.harvard.edu/
gking.harvard.edu/
isites.harvard.edu/
...

There’s many possible ways to sort, though. So I took my WWW archive as of 15 June 2014, optimized all PNGs & JPEGs with optipng & jpegoptim, ran all the files through filter-urls & deleted the ones which failed (this took out all of the JS files, which is fine since I don’t think those are useful for archival purposes), and was left with ~86.5GB of files. Then I tested out several ways of sorting the filenames to see what gave the best compression on my corpus.

First, I establish the baseline:

1. Size of uncompressed unsorted tarball, which eliminates filesystem overhead and tells us how much compression is really saving:

   cd ~/www/ && find ./ -type f -print0 | tar c --to-stdout --no-recursion --null --files-from - | wc --bytes
   86469734400
   1x

2. Size of sorted tarball:

   cd ~/www/ && find . -type f -print0 | sort --zero-terminated | \
                tar c --to-stdout --no-recursion --null --files-from - | wc --bytes
   86469734400
   1x

   So sorting a tarball doesn’t give any benefits. This is mostly as I expected, since tar is only supposed to produce a linear archive packing together all the specified files and otherwise preserve them exactly. I thought there might have been some sort of consolidation of full path-names which might yield a small space savings, but apparently not.

Now we can begin sorting before compression. I thought of 6 approaches; in decreasing order of final archive (smaller=better):

1. Sort by file names, simply by reversing, sorting, unreverse (foo.png ... bar.png reverses to gnp.oof ... gnp.rab, which then sort together, and then losslessly reverse back to bar.png / foo.png / ...):

   cd ~/www/ && find . -type f | rev | sort | rev | \
                tar c --to-stdout --no-recursion --files-from - | xz -9 --stdout | wc --bytes
   24970605748
   0.2887x

   (Note that find+rev doesn’t correctly handle filenames with the wrong/non-UTF-8 encoding; I ultimately used brute force in the form of detox to find all the non-UTF-8 files and rename them.)

2. Compress tarball, but without any sorting (files are consumed in the order find produces them in the filesystem):

   cd ~/www/ && find . -type f -print0 | \
                tar c --to-stdout --no-recursion --null --files-from - | xz -9 --stdout | wc --bytes
   24268747400
   0.2806x

3. Sort by file suffixes, trying to parsing the filenames first:

   cd ~/www/ && find . -type f -printf '%f/%p\n' | sort --field-separator="." --key=2  | cut -f2- -d/ | \
                tar c --to-stdout --no-recursion --files-from - | xz -9 --stdout | wc --bytes
   24097155132
   0.2786x

4. Sort normally, in lexicographic order (subdomain, domain, TLD, subdirectories & files etc):

   cd ~/www/ && find . -type f -print0 | sort --zero-terminated | \
       tar c --to-stdout --no-recursion --null --files-from - | xz -9 --stdout | wc --bytes
   23967317664
   0.2771x

5. Sort by middle of domain:

   cd ~/www/ && find . -type f -print0 | sort --zero-terminated --key=3 --field-separator="." | \
                 tar c --to-stdout --no-recursion --null --files-from - | xz -9 --stdout | wc --bytes
   23946061272
   0.2769x

6. Sort by first subdirectory (if there’s a bunch of foo.com/wp-content/* & bar.com/wp-content/* files, then the /wp-content/ files will all sort together regardless of f and b being far from each other; similarly for domain.com/images/, domain.com/css/ etc):

   cd ~/www/ && find . -type f -print0 | sort --zero-terminated --key=3 --field-separator="/"  | \
                tar c --to-stdout --no-recursion --null --files-from - | xz -9 --stdout | wc --bytes
   23897682908
   0.2763x

Surprisingly, #1, reversing filenames in order to sort on the suffixes, turns out to be even worse than not sorting at all. The improved attempt to sort on filetypes doesn’t do much better, although it at least beats the baseline of no-sorting; it may be that to get a compression win real semantic knowledge of filetypes will be needed (perhaps calling file on every file and sorting by the detected filetype?). The regular sort also performs surprisingly well, but my intuitions win for once and it’s beaten by my previous strategy of sorting by the middle of domains. Finally, the winner is a bit of a surprise too, a sort I only threw in out of curiosity because I noticed blogs tend to have similar site layouts.

In this case, the best version saved 24268747400−23897682908=371064492$24268747400-23897682908=371064492$ or 371MB over the unsorted version. Not as impressive as in the next use case, but enough to show this seems like a real gain

Separate mirrors

Top-level domains are not the only thing we might want to sort differently on. To take my mirrors of darknet market drug sites such as Silk Road 1: I download a site each time as a separate wget run in a timestamped folder. So in my Silk Road 2 folder, I have both 2013-12-25/ & 2014-01-15/. These share many similar & identical files so they compress together with xz down from 1.8GB to 0.3GB.

But they could compress even better: the similar files may be thousands of files and hundreds of megabytes away by alphabetical or file-inode order, so even with a very large window and a top-notch compressor, it will fail to spot many long-range redundancies. In between 2013-12-25/default.css and 2014-01-15/default.css is going to be all sorts of files which have nothing to do with CSS, like 2014-01-16/items/2-grams-of-pure-vanilla-ketamine-powder-dutchdope?vendor_feedback_page=5 and 2014-01-16/items/10-generic-percocet-10-325-plus-1-30mg-morphine. You see the problem.

Because we sort the files by all files starting with 2013 and then all files starting 2014, we lose all proximity. If instead, we could sort by subfolder and only then by the top-level folder, then we’d have everything line up nicely. Fortunately, we already know how to do this! Reuse the sort-key trick, specify / as the delimiter to parse on, and the nth field to sort on. We feed it a file list, tell it to break filenames by /, and then to sort on a lower level, and if we did it right, we will indeed get output like 2013-12-25/default.css just before 2014-01-15/default.css, which will do wonders for our compression, and which will pay ever more dividends as we accumulate more partially-redundant mirrors.

Here is an example of output for my Pandora mirrors, where, due to frequent rumors of its demise triggering mirroring on my part, I have 5 full mirrors at the time of testing; and naturally, if we employ the sort-key trick (find . -type f | sort --key=3 --field-separator="/"), we find a lot of similar-sounding files:

./2014-01-15/profile/5a66e5238421f0422706b267b735d2df/6
./2014-01-16/profile/5a9df4f5482d55fb5a8997c270a1e22d
./2013-12-25/profile/5a9df4f5482d55fb5a8997c270a1e22d/1
./2014-01-15/profile/5a9df4f5482d55fb5a8997c270a1e22d.1
./2013-12-25/profile/5a9df4f5482d55fb5a8997c270a1e22d/2
./2014-01-15/profile/5a9df4f5482d55fb5a8997c270a1e22d.2
./2013-12-25/profile/5a9df4f5482d55fb5a8997c270a1e22d/3
./2014-01-15/profile/5a9df4f5482d55fb5a8997c270a1e22d/4
./2014-01-15/profile/5a9df4f5482d55fb5a8997c270a1e22d/5
./2014-01-15/profile/5a9df4f5482d55fb5a8997c270a1e22d/6
./2013-12-25/profile/5abb81db167294478a23ca110284c587
./2013-12-25/profile/5acc44d370e305e252dd4e2b91fda9d0/1
./2014-01-15/profile/5acc44d370e305e252dd4e2b91fda9d0.1
./2013-12-25/profile/5acc44d370e305e252dd4e2b91fda9d0/2
./2014-01-15/profile/5acc44d370e305e252dd4e2b91fda9d0.2

Note the interleaving of 5 different mirrors, impossible in a normal left-to-right alphabetical sort. You can bet that these 4 files (in 15 versions) are going to compress much better than if they were separated by a few thousand other profile pages.

So here’s an example invocation (doing everything in pipelines to avoid disk IO):

find . -type f -print0 | sort --zero-terminated --key=3 --field-separator="/"
 | tar --no-recursion --null --files-from - -c
 | xz -9 --extreme --stdout
 > ../mirror.tar.xz

Used on my two Silk Road 2 mirrors which together weigh 1800M, a normal run without the --key/--field-separator options, as mentioned before, yields a 308M archive. That’s not too bad. Certainly much better than hauling around almost 2GB. However - if I switch to the sort-key trick, however, the archive is now 271M, or 37M less. Same compression algorithm, same files, same unpacked results, same speed, just 2 little obscure sort options… and I get an archive 87% the size of the original.

Not impressed? Well, I did say that the advantage increases with the number of mirrors to extract redundancy from. With only 2 mirrors, the SR2 results can’t be too impressive. How about the Pandora mirrors? 5 of them gives the technique more scope to shine. And as expected, it’s even more impressive when I compare the Pandora archives: 71M vs 162M. The sort-keyed archive saves 56% of the regular archive’s size. The final Pandora archive, released as part of my DNM archive, boasts 105 mirrors from 2013-12-25 to 2014-11-05 of 1,745,778 files (many duplicates due to being present over multiple crawls) taking up 32GB uncompressed. This archive is 0.58GB (606,203,320 bytes) when sort-keyed; when sorted alphabetically, it is instead 7.5GB (7,466,374,836 bytes) or 12.31x larger, so the sort-keyed archive saves 92%. Forum mirrors are even more redundant than market mirrors because all content added to a forum will be present in all subsequent crawls, so each crawl will yield the previous crawl plus some additional posts; hence, the Pandora forums’s raw size of 8,894,521,228 bytes shrinks to 283,747,148 bytes when compressed alphabetically but shrinks dramatically to the 31x smaller sort-keyed archive of 9,084,508 bytes, saving 97%!

Clearly in these DNM use-cases, sort-keying has gone from a cute trick to indispensable.

Alternatives

The sort-key trick is most useful when we can infer a lot of spatial locality just from parts of the file names, it’s not a panacea. There are other approaches:

1. if we have multiple temporally-ordered datasets and we don’t mind making it more difficult to access older copies, it may be simpler to store the data as a DVCS like git where each dataset is a large patch

2. sort files by minimizing binary differences between them using Timm S. Müller’s binsort utility

   The default optimization setting of -o15 underperforms the sort-key trick on both the SR2 and Pandora datasets by 5-10MB, and a higher setting like -o1000 is best. Note that binsort is O(n2)$O(n^2)$ in number of files, so it’s limited to sorting <10,000 files. An example pipeline for compressing posts from the Silk Road forums regarding a minor Ponzi scheme there, since the filenames offer little guidance for a semantically-meaningful sort:

   find dkn255hz262ypmii.onion/ -type f -exec fgrep -i -l vladimir  {} \; >> ponzi.txt
   find dkn255hz262ypmii.onion/ -type f -exec fgrep -i -l Limetless {} \; >> ponzi.txt
   
   mkdir ponzi/ && cp `cat ponzi.txt` ponzi/
   ~/bin/binsort/binsort -t 6 -o 1000 ponzi/ > ponzi-list.txt
   cat ponzi-list.txt | tar --no-recursion --files-from - -c | xz -9 --extreme --stdout > ~/srf1-ponzi.tar.xz

   (Embarrassingly, binsort seems to underperform on this dataset: the files are 380M uncompressed, 3.536M sort-keyed, and 3.450M sorted.)

3. if we know there are many duplicates but they are far apart by a lexicographic sort and we don’t have any good way to sort filenames and a lot of RAM, we can try out a compression tool which specifically looks for long-range redundancies throughout the entire bitstream, such as lrzip (homepage/Arch wiki). lrzip is packaged in Debian and should be at least as good as xz since it too uses LZMA; but it is an obscure tool which is not a default install on Linux distributions like xz is, and this makes distributing archives to other people difficult.

External links

• Sorting multiple keys with Unix sort
• SimHash: Hash-based Similarity Detection (see also MinHash)