Search

Searching

GS: if no full­text in upper right, look for soft walls

Hav­ing launched your search in, pre­sum­ably, Google Scholar, you must nav­i­gate the GS re­sults. For GS, it is often as sim­ple as click­ing on the [PDF] or [HTML] link in the top right which de­notes (what GS be­lieves to be) a full­text link, eg:

GS: if no full­text in upper right, look for soft walls. In GS, re­mem­ber that a full­text link is not al­ways de­noted by a “[PDF]” link! Check the top hits by hand: there are often ‘soft walls’ which block web spi­ders but still let you down­load full­text (per­haps after sub­stan­tial has­sle, like SSRN).

Note that GS sup­ports other use­ful fea­tures like alerts for search queries, alerts for any­thing cit­ing a spe­cific paper, and re­verse ci­ta­tion searches (to fol­lowup on a paper to look for failures-to-replicate or crit­i­cisms of it).

Drilling Down

A use­ful hit may not turn up im­me­di­ately. Life is like that.⁠^⁠3⁠ You may have to get cre­ative:

• Title searches: if a paper full­text doesn’t turn up on the first page, start tweak­ing (hard rules can­not be given for this, it re­quires de­vel­op­ment of “me­chan­i­cal sym­pa­thy”⁠ and ask­ing a mix­ture of “how would a ma­chine think to clas­sify this” and “how would other peo­ple think to write this”):

• The Golden Mean: Keep mind when search­ing, you want some but not too many or too few re­sults. A few hun­dred hits in GS is around the sweet spot. If you have less than a page of hits, you have made your search too spe­cific.

  If noth­ing is turn­ing up, try trim­ming the title. Ti­tles tend to have more er­rors to­wards the end than the be­gin­ning, and peo­ple often drop words or phrases at the end—es­pe­cially sub­ti­tles. So start cut­ting words off the end of the title to broaden the search. Think about what kinds of er­rors you make when you re­call ti­tles: you drop punc­tu­a­tion or sub­ti­tles, sub­sti­tute in more fa­mil­iar syn­onyms, or oth­er­wise sim­plify it. (How might OCR soft­ware screw up a title?)

  Pay at­ten­tion to tech­ni­cal terms that pop up in as­so­ci­a­tion with your own query terms, par­tic­u­larly in the snip­pets or full ab­stracts. Which ones look like they might be more pop­u­lar than yours, or in­di­cate yours are usu­ally used slightly dif­fer­ent from you think they mean? You may need to switch terms.

  If delet­ing a few terms then yields way too many hits, try to fil­ter out large classes of hits with a nega­tion foo -bar, adding as many as nec­es­sary; also use­ful is using OR clauses to open up the search in a more re­stricted way by adding in pos­si­ble syn­onyms, with paren­the­ses for group. This can get quite elab­o­rate, and bor­der on hack­ing⁠—I have on oc­ca­sion re­sorted to search queries as baroque as (foo OR baz) AND (qux OR quux) -bar -garply -waldo -fred to the point where I hit search query length lim­its and CAPTCHA bar­ri­ers.⁠^⁠4⁠ (By that point, it is time to con­sider al­ter­nate at­tacks.)

• Tweak The Title: quote the title; delete any sub­ti­tle; try the sub­ti­tle in­stead; be sus­pi­cious of any char­ac­ter which is not al­phanu­meric and if there are colons, split it into two title quotes (in­stead of search­ing Foo bar: baz quux, or "Foo bar: baz quux", search "Foo bar" "baz quux"); swap their order.

• Tweak The Meta­data:

  • Add/re­move the year.

  • Add/re­move the first au­thor’s sur­name. Try search­ing GS for just the au­thor (author:foo).

• Delete Odd Char­ac­ters/Punc­tu­a­tion:

  Lib­gen had trou­ble with colons for a long time, and many web­sites still do (eg. GoodReads⁠); I don’t know why colons in par­tic­u­lar are such trou­ble, al­though hy­phens/em-dashes and any kind of quote or apos­tro­phe or pe­riod are prob­lem­atic too. Watch out for words which may be space-separated—if you want to find Arpad Elo’s⁠ epochal The Rat­ing of Chess­play­ers⁠ in Lib­gen, you need to search “The Rat­ing of Chess Play­ers” in­stead! (This is also an ex­am­ple of why falling back to search by au­thor is a good idea.)

• Tweak Spelling: Try al­ter­nate spellings of British/Amer­i­can terms. This shouldn’t be nec­es­sary, but then, delet­ing colons or punc­tu­a­tion shouldn’t be nec­es­sary ei­ther.

• Check For Book Pub­li­ca­tion: many pa­pers are pub­lished in the form of book an­tholo­gies, not jour­nal ar­ti­cles. So look for the book if the paper is mys­te­ri­ously ab­sent.

  A book will not nec­es­sar­ily turn up in GS and thus its con­stituent pa­pers may not ei­ther; sim­i­larly, while SH does a good job of link­ing ar­ti­cle pay­walls to their re­spec­tive book com­pi­la­tion in LG, it is far from in­fal­li­ble. If a paper was pub­lished in any kind of ‘pro­ceed­ing’ or ‘con­fer­ence’ or ‘se­ries’ or any­thing with an ISBN, the paper may be ab­sent from the usual places but the book read­ily avail­able. It can be quite frus­trat­ing to be search­ing hard for a paper and re­al­ize the book was there in plain sight all along. (My sug­ges­tion in such cases for ⁠post-finding⁠ is to cut out the rel­e­vant page range & up­load the paper for oth­ers to more eas­ily find.)

• Use URLs: if you have a URL, try search­ing chunks of it, typ­i­cally to­wards the end, strip­ping out dates and do­main names.

• Date Search:

  Use a search en­gine (eg. G/GS) date range fea­ture (in “Tools”) to search ±4 years: meta­data can be wrong, pub­lish­ing con­ven­tions can be odd (eg. a mag­a­zine pub­lished in ‘June’ may ac­tu­ally be pub­lished sev­eral months be­fore or after), pub­lish­ers can be ex­tremely slow. This is par­tic­u­larly use­ful if you add a date con­straint & si­mul­ta­ne­ously loosen the search query to turn up the most temporally-relevant of what would oth­er­wise be far too many hits. If this doesn’t turn up the rel­e­vant tar­get, it might turn up re­lated dis­cus­sions or fixed ci­ta­tions, since most things are cited most shortly after pub­li­ca­tion and then van­ish into ob­scu­rity.

  

  

  If a year is not spec­i­fied, try to guess from the medium: pop­u­lar media has heavy re­cen­tist bias & prefers only con­tem­po­rary re­search which is ‘news’, while aca­d­e­mic pub­li­ca­tions go back a few more years; the style of the ref­er­ence can give a hint as to how rel­a­tively old some men­tioned re­search or writ­ings is. Fre­quently, given the au­thor sur­name and a rea­son­able guess at some re­search being a year or two old, the name + date-range + key­word in GS will be enough to find the paper.

  • Con­sider er­rors: typos are com­mon. If noth­ing is show­ing up in the date-range de­spite a spe­cific date, per­haps there was a ty­po­graphic error. Even a dili­gent typ­ist will oc­ca­sion­ally copy meta­data from a pre­vi­ous entry or type the same char­ac­ter twice or 2 char­ac­ters in the wrong order, and for num­bers, there is no spellcheck to help catch such er­rors. Au­thors fre­quently prop­a­gate bib­li­o­graphic er­rors⁠ with­out cor­rect­ing them (demon­strat­ing, in­ci­den­tally, that they prob­a­bly did not read the orig­i­nal and so any sum­maries should be taken with a grain of salt). Think about plau­si­ble trans­po­si­tions & neigh­bor­ing keys on a QW­ERTY key­board: eg. a year like “1976” may ac­tu­ally be 1966_60ya, 1967_59ya, 1975_51ya, 1977_49ya, or 1986_40ya (but it will not be 1876_150ya or 2976 or 1987_39ya). What typos would you make if you were read­ing or typ­ing in a hurry? What OCR er­rors are likely, such as con­fus­ing ‘3’/‘8’?

• Add Jar­gon: Add tech­ni­cal ter­mi­nol­ogy which might be used by rel­e­vant pa­pers; for ex­am­ple, if you are look­ing for an ar­ti­cle on col­lege ad­mis­sions sta­tis­tics, any such analy­sis would prob­a­bly be using lo­gis­tic re­gres­sion⁠ and, even if they do not say “lo­gis­tic re­gres­sion” (in favor of some more pre­cise yet unguess­able term) would ex­press their ef­fects in terms of “odds”.

  If you don’t know what jar­gon might be used, you may need to back off and look for a re­view ar­ti­cle or text­book or WP page and spend some qual­ity time read­ing. If you’re using the wrong term, pe­riod, noth­ing will help you; you can spend hours going through count­less pages, but that won’t make the wrong term work. You may need to read through overviews until you fi­nally rec­og­nize the skele­ton of what you want under a com­pletely dif­fer­ent (and often rather ob­tuse) name. Noth­ing is more frus­trat­ing that know­ing there must be a large lit­er­a­ture on a topic (“Cowen’s Law”) but being un­able to find it be­cause it’s named some­thing com­pletely dif­fer­ent from ex­pected—and many fields have dif­fer­ent names for the same con­cept or tool. (Oc­ca­sion­ally peo­ple com­pile “Rosetta stones” to trans­late be­tween fields: eg. Baez & Stay2009⁠, Bert­sekas2018⁠, ⁠Metz et al 2018’s Table 1⁠. These are in­valu­able.)

• Even The Hum­ble Have A Tale: Be­ware hastily dis­miss­ing ‘bib­li­o­graphic’ web­sites as use­less—they may have more than you think.

  While a bibliographic-focused li­brary site like elibrary.ru is (al­most) al­ways use­less & clut­ters up search re­sults by host­ing only the ci­ta­tion meta­data but not full­text, every so often I run into a pe­cu­liar for­eign web­site (often In­dian or Chi­nese) which hap­pens to have a scan of a book or paper. (eg. Dar­ling­ton1954⁠, which eluded me for well over half an hour until, tak­ing the al­ter­nate ap­proach of hunt­ing its vol­ume, I out of des­per­a­tion clicked on an In­dian index/li­brary web­site which… had it. Go fig­ure.) Some­times you have to check every hit, just in case.

• Search In­ter­net Archive:

  The In­ter­net Archive (IA) de­serves spe­cial men­tion as a tar­get be­cause it is the In­ter­net’s attic, burst­ing at the seams with a re­mark­able as­sort­ment of scans & up­loads from all sorts of sources—not just archiv­ing web pages, but scan­ning uni­ver­sity col­lec­tions, ac­cept­ing up­loads from rogue archivists and hack­ers and ob­ses­sive fans and the afore­men­tioned In­dian/Chi­nese li­braries with more laissez-faire ap­proaches.⁠^⁠5⁠ This ex­tends to its media col­lec­tions as well—who would ex­pect to find so many old science-fiction mag­a­zines (as well as ⁠many other mag­a­zines⁠), a near-infinite num­ber of Grate­ful Dead⁠ record­ings, the ⁠orig­i­nal 114 episodes⁠ of Tom and Jerry⁠, or thou­sands of ⁠ar­cade⁠ & ⁠con­sole⁠ & ⁠PC⁠ & ⁠Flash⁠ games (all playable in-browser)? The In­ter­net Archive is a ver­i­ta­ble In­ter­net in and of it­self; the prob­lem, of course, is find­ing any­thing…

  So not in­fre­quently, a book may be avail­able, or a paper ex­ists in the mid­dle of a scan of an en­tire jour­nal vol­ume, but the IA will be ranked very low in search queries and the snip­pet will be mis­lead­ing due to bad OCR. A good search strat­egy is to drop the quotes around ti­tles or ex­cerpts and focus down to site:archive.org and check the first few hits by hand. (You can also try the rel­a­tively new ⁠“In­ter­net Archive Scholar”⁠, which ap­pears to be more com­pre­hen­sive than Google-site-search.)

Hard Cases

If the basic tricks aren’t giv­ing any hints of work­ing, you will have to get se­ri­ous. The title may be com­pletely wrong, or it may be in­dexed under a dif­fer­ent au­thor, or not di­rectly in­dexed at all, or hid­den in­side a data­base. Here are some in­di­rect ap­proaches to find­ing ar­ti­cles:

• Re­verse Ci­ta­tions: Take a look in GS’s “re­lated ar­ti­cles” or “cited by” to find sim­i­lar ar­ti­cles such as later ver­sions of a paper which may be use­ful. (These are also good fea­tures to know about if you want to check things like “has this ever been repli­cated?” or are still fig­ur­ing out the right jar­gon to search.)

• Anom­alous Hits: Look for hints of hid­den bib­li­o­graphic con­nec­tions and anom­alous hits.

  Does a paper pop up high in the search re­sults which doesn’t seem to make sense, such as not con­tain­ing your key­words in the dis­played snip­pet? GS gen­er­ally pe­nal­izes items which exist as sim­ply bib­li­o­graphic en­tries, so if one is ranked high in a sea of full­texts, that should make you won­der why it is being pri­or­i­tized. Sim­i­larly, for Google Books (GB): a book might be for­bid­den from dis­play­ing even snip­pets but rank high; that might be for a good rea­son—it may ac­tu­ally con­tain the full­text hid­den in­side it, or some­thing else rel­e­vant.

  Like­wise, you can­not trust meta­data too much. The in­ferred or claimed title may be wrong, and a hit may be your de­sired tar­get lurk­ing in dis­guise.

• Com­pi­la­tion Files: Some pa­pers can be found by search­ing for the vol­ume or book title to find it in­di­rectly, es­pe­cially con­fer­ence pro­ceed­ings or an­tholo­gies; many pa­pers ap­pear to not be avail­able on­line but are merely buried deep in­side a 500-page PDF, and the G snip­pet list­ing is mis­lead­ing.

  Con­fer­ences are par­tic­u­larly com­plex bib­li­o­graph­i­cally, so you may need to apply the same tricks as for page ti­tles: drop parts, don’t fix­ate on the num­bers, know that the au­thors or ISBN or or­der­ing of “title: sub­ti­tle” can dif­fer be­tween sources, etc.

  PDF Pages Are Link­able

  Tech trick: you can link to a spe­cific page num­ber N of any PDF by adding #page=N to the URL⁠ (eg. this link⁠ links to the text sam­ples in the Mega­tron⁠ paper on page 13, rather than the first page)

• Search By Issue: An­other ap­proach is to look up the list­ing for a jour­nal issue, and find the paper by hand; some­times pa­pers are listed in the jour­nal issue’s on­line Table of Con­tents, but just don’t ap­pear in search en­gines (‽). In par­tic­u­larly in­sid­i­ous cases, a paper may be dig­i­tized & avail­able—but lumped in with an­other paper due to error, or only as part of a catch-all file which con­tains the last 20 mis­cel­la­neous pages of an issue. Page range ci­ta­tions are par­tic­u­larly help­ful here be­cause they show where the over­lap is, so you can down­load the sus­pi­cious over­lap­ping ‘pa­pers’ to see what they re­ally con­tain.

  Es­o­teric as this may sound, this has been a prob­lem on mul­ti­ple oc­ca­sions. I searched in vain for any hint of Shep­ard1929’s⁠ ex­is­tence, half-convinced it was a typo for his 1959 pub­li­ca­tion, until I turned to the raw jour­nal scans. A par­tic­u­larly epic ex­am­ple was Shock­ley1966⁠ where after an hour of hunt­ing, all I had was bib­li­o­graphic echoes de­spite ap­par­ently being pub­lished in a high-profile, eas­ily ob­tained, & def­i­nitely dig­i­tized jour­nal, Sci­ence—leav­ing me thor­oughly baf­fled. I even­tu­ally looked up the ToC and in­ferred it had been hid­den in a set of ab­stracts!⁠^⁠6⁠ (One symp­tom of the ‘ab­stract’ or ‘con­fer­ence pre­sen­ta­tion’ prob­lem is when the aca­d­e­mic data­bases keep claim­ing to have full­text of a paper, but then mys­te­ri­ously error out when you try to ac­tu­ally ac­cess it. Ap­par­ently they can be in­con­sis­tent in­ter­nally and know that they have it in the full jour­nal scans, but not know where just that one ab­stract ‘is’.) Or a num­ber of SMPY⁠ pa­pers turned out to be split or merged with neigh­bor­ing items in jour­nal is­sues, and I had to fix them by hand.

• Mas­ters/PhD The­ses: sorry. It may be hope­less if it’s pre-2000. You may well find the ci­ta­tion and even an ab­stract, but ac­tual full­text…?

  If you have a uni­ver­sity proxy, you may be able to get a copy off Pro­Quest⁠ (spe­cial­iz­ing in US the­ses). If Pro­Quest does not allow a down­load but in­dexes it, that usu­ally means it has a copy archived on mi­cro­film/mi­cro­fiche, but no one has yet paid for a scan to be made; you can sign up with­out any spe­cial per­mis­sion, and then pur­chase Pro­Quest scans for ~$43 (as of 2023), and that gives you a down­load­able PDF. (They ap­par­ently scan non-digital works from their vast back­log only on re­quest, so it’s al­most like ran­som­ing pa­pers; which means that buy­ing a scan makes it avail­able to aca­d­e­mic sub­scribers as part of the Pro­Quest data­base.)

  Oth­er­wise, you need full uni­ver­sity ILL ser­vices⁠^⁠7⁠, and even that might not be enough (a sur­pris­ing num­ber of uni­ver­si­ties ap­pear to re­strict ac­cess only to the uni­ver­sity stu­dents/fac­ulty, with the com­pli­cat­ing fac­tor of most the­ses being stored on mi­cro­film).

• Re­verse Image Search⁠: If im­ages are in­volved, a re­verse image search in Google Im­ages or Tin­Eye or Yan­dex Search⁠ can turn up im­por­tant leads.

  Belling­cat⁠ has a good guide by Aric Toller: “Guide To Using Re­verse Image Search For In­ves­ti­ga­tions”. (Yan­dex image search ap­pears to ex­ploit face recog­ni­tion, text OCR, and other ca­pa­bil­i­ties Google Im­ages will not, and bows less to copy­right con­cerns.)

  Use Browser Page Info to By­pass Image Re­stric­tions

  If you are hav­ing trou­ble down­load­ing an image from a web page which is badly/ma­li­ciously de­signed to stop you, use “View Page Info”’s⁠ (C-I) “Media” tab (⁠eg⁠), which will list the im­ages in a page and let one down­load them di­rectly.

• Enemy Ac­tion: Is a page or topic not turn­ing up in Google/IA that you know ought to be there? Check the web­site’s robots.txt⁠ & sitemap⁠. While not as rel­e­vant as they used to be (due to in­creas­ing use of dy­namic pages & en­ti­ties ig­nor­ing it), robots.txt can some­times be rel­e­vant: key URLs may be ex­cluded from search re­sults, and overly-restrictive robots.txt can cause enor­mous holes in IA cov­er­age, which may be im­pos­si­ble to fix (but at least you’ll know why).

• Pa­tience: not every pay­wall can be by­passed im­me­di­ately, and pa­pers may be em­bar­goed or prox­ies not im­me­di­ately avail­able.

  If some­thing is not avail­able at the mo­ment, it may be­come avail­able in a few months. Use cal­en­dar re­minders to check back in to see if an em­bar­goed paper is avail­able or if LG/SH have ob­tained it, and whether to pro­ceed to ad­di­tional search steps like man­ual re­quests.

• Do­main Knowledge-Specific Tips:

  • Twit­ter: Twit­ter is in­dexed in Google so web searches may turn up hits, but if you know any meta­data, Twit­ter’s na­tive search func­tions are still rel­a­tively pow­er­ful (al­though Twit­ter lim­its searches in many ways in order to drive busi­ness to its staggeringly-expensive ‘fire­hose’ & his­tor­i­cal an­a­lyt­ics). Use of Twit­ter’s “ad­vanced search”⁠ in­ter­face, par­tic­u­larly the from: & to: search query op­er­a­tors⁠, can vastly cut down the search space. (Also of note: list:, -filter:retweets, near:, url:, & since:/until:.)

  • US fed­eral courts: US fed­eral court doc­u­ments can be down­loaded off PACER⁠ after reg­is­tra­tion.

    PACER is pay-per-page ($0.13^$0.10₂₀₁₈/page) but users under a cer­tain level each quar­ter (cur­rently $20^$15₂₀₁₈) have their fees waived, so if you are care­ful, you may not need to pay any­thing at all. There is a pub­lic mir­ror on Courtlis­tener, called RECAP⁠, which can be freely searched & down­loaded.

    If you fail to find a case in RECAP and must use PACER (as often hap­pens for ob­scure cases), please in­stall the Fire­fox/Chrome RECAP browser ex­ten­sion⁠, which will copy any­thing you down­load into RECAP. (This can be handy if you re­al­ize later that you should’ve kept a long PDF you down­loaded or want to double-check a docket.)

    Nav­i­gat­ing PACER can be dif­fi­cult be­cause it is an old & highly spe­cial­ized com­puter sys­tem which as­sumes you are a lawyer, or at least very fa­mil­iar with PACER & the Amer­i­can fed­eral court sys­tem. As a rule of thumb, if you are look­ing up a par­tic­u­lar case, what you want to do is to search for the first name & sur­name (even if you have the case ID) for ei­ther crim­i­nal or civil cases as rel­e­vant, and pull up all cases which might per­tain to an in­di­vid­ual; there can be mul­ti­ple cases, cases can hi­ber­nate for years, be closed, re­opened as a dif­fer­ent case num­ber, etc. Once you have found the most ac­tive or rel­e­vant case, you want to look at the “docket”, and check the op­tions to see all doc­u­ments in the case. This will pull up a list of many doc­u­ments as the case un­folds over time; most of these doc­u­ments are legal bu­reau­cracy, like resched­ul­ing hear­ings or no­ti­fi­ca­tions of changed lawyers. You want the longest doc­u­ments, as those are most likely to be use­ful. In par­tic­u­lar, you want the “in­dict­ment”, the “crim­i­nal com­plaint”⁠^⁠8⁠, and any tran­scripts of trial tes­ti­mony.⁠^⁠9⁠ Shorter doc­u­ments, like 1–2pg en­tries in the docket, can be use­ful, but are much less likely to be use­ful un­less you are in­ter­ested in the exact de­tails of how things like pre-trial ne­go­ti­a­tions un­fold. So care­lessly choos­ing the ‘down­load all’ op­tion on PACER may blow through your quar­terly bud­get with­out get­ting you any­thing in­ter­est­ing (and also may in­ter­fere with RECAP up­load­ing doc­u­ments).

    There is no equiv­a­lent for state or county court sys­tems, which are balka­nized and use a thou­sand dif­fer­ent sys­tems (often pri­va­tized & charg­ing far more than PACER); those must be han­dled on a case by case basis. (In­ter­est­ing trivia point: ac­cord­ing to Nick Bil­ton’s ac­count of the Silk Road 1 case, the FBI and other fed­eral agen­cies in the SR1 in­ves­ti­ga­tion would de­lib­er­ately steer cases into state rather than fed­eral courts in order to hide them from the rel­a­tive trans­parency of the PACER sys­tem. The use of mul­ti­ple court sys­tems can back­fire on them, how­ever, as in the case of SR2’s Doc­tor­Clu (see the DNM ar­rest cen­sus⁠ for de­tails), where the local po­lice fil­ings re­vealed the use of hack­ing tech­niques to deanonymize SR2 Tor⁠ users, im­pli­cat­ing CMU’s CERT cen­ter—de­tails which were be­lat­edly scrubbed from the PACER fil­ings.)

  • char­ity fi­nan­cials: for USA char­ity fi­nan­cial fil­ings, do Form 990 site:charity.com and then check GuideStar⁠ (eg. look­ing at Girl Scouts fil­ings⁠ or ⁠“Case Study: Read­ing Edge’s fi­nan­cial fil­ings”⁠). For UK char­i­ties, the Char­ity Com­mis­sion for Eng­land and Wales⁠ may be help­ful.

  • ed­u­ca­tion re­search: for any­thing re­lated to ed­u­ca­tion, do a site search of ERIC⁠, which is sim­i­lar to IA in that it will often have full­text which is buried in the usual search re­sults

  • Well­come Li­brary: the Well­come Li­brary has many old jour­nals or books dig­i­tized which are im­pos­si­ble to find else­where; un­for­tu­nately, their SEO is awful & their PDFs are un­nec­es­sar­ily hid­den be­hind click-through EULAs, so they will not show up nor­mally in Google Scholar or else­where. If you see the Well­come Li­brary in your Google hits, check it out care­fully.

  • mag­a­zines (as op­posed to schol­arly or trade jour­nals) are hard to get.

    They are not cov­ered in Lib­gen/Sci-Hub, which out­source that to MagzDB; cov­er­age is poor, how­ever. An al­ter­na­tive is pdf-giant⁠. Par­tic­u­larly for pre-2000 mag­a­zines, one may have to re­sort to look­ing for old used copies on eBay. Some mag­a­zines are eas­ier than oth­ers—I gen­er­ally give up if I run into a New Sci­en­tist ci­ta­tion be­cause it’s never worth the trou­ble.

• News­pa­pers: like the­ses, tricky. I don’t know of any gen­eral so­lu­tions short of a Lex­is­Nexis sub­scrip­tion.⁠^⁠10⁠ An in­ter­est­ing re­source for Amer­i­can pa­pers is ⁠Chron­i­cling Amer­ica’s “His­toric Amer­i­can News­pa­per” scans.

By Quote or Description

For quote/de­scrip­tion searches: if you don’t have a title and are falling back on search­ing quotes, try vary­ing your search sim­i­larly to ti­tles:

• Novel sen­tences: Try the easy search first—what­ever looks most mem­o­rable or unique.

• Short quotes are unique: Don’t search too long a quote, a sen­tence or two is usu­ally enough to be near-unique, and can be help­ful in turn­ing up other sources quot­ing dif­fer­ent chunks which may have bet­ter ci­ta­tions.

  • Break up quotes: Be­cause even phrases can be unique, try mul­ti­ple sub-quotes from a big quote, es­pe­cially from the be­gin­ning and end, which are likely to over­lap with quotes which have prior or sub­se­quent pas­sages. This can be crit­i­cal with apoc­ryphal quotes, which often delete less witty sub-passages while ac­cret­ing new ma­te­r­ial. (An ex­treme ex­am­ple would be the ⁠Oliver Heav­i­side quote⁠.)

  • Odd idio­syn­cratic word­ing: Search for oddly-specific phrases or words, es­pe­cially num­bers. 3 or 4 key­words is usu­ally enough.

  • Para­phras­ing: Look for pas­sages in the orig­i­nal text which seem like they might be based on the same source, par­tic­u­larly if they are sim­ply dropped in with­out any hint at sourc­ing and don’t sound like the au­thor; au­thors typ­i­cally don’t cite every time they draw on a source, usu­ally only the first time, and dur­ing edit­ing the ‘first’ ap­pear­ance of a source could eas­ily have been moved to later in the text. All of these ad­di­tional uses are some­thing to add to your searches.

• Ro­bust Quotes: You are fight­ing a game of Chi­nese whis­pers, so look for unique-sounding sen­tences and terms which can sur­vive gar­bling in the re­peated trans­mis­sions.

  Mem­o­ries are urban leg­ends told by one neu­ron to an­other over the years. Pay at­ten­tion to how you mis-remember things: you dis­tort them by sim­pli­fy­ing them, round­ing them to the near­est eas­i­est ver­sion, and by adding in de­tails which should have been there. Avoid phrases which could be eas­ily re­worded in mul­ti­ple equiv­a­lent ways, as peo­ple usu­ally will re­word them when quot­ing from mem­ory, screw­ing up lit­eral searches. Re­mem­ber the fal­li­bil­ity of mem­ory and the basic prin­ci­ples of tex­tual crit­i­cism⁠: peo­ple sub­sti­tute easy-to-remember ver­sions for the hard⁠, long⁠^⁠11⁠, or un­usual orig­i­nal.

• Tweak Spelling: Watch out for punc­tu­a­tion and spelling dif­fer­ences hid­ing hits.

• Gra­di­ent As­cent: Longer, less witty ver­sions are usu­ally closer to the orig­i­nal and a sign you are on the right trail. The worse, the bet­ter. Sniff in the di­rec­tion of worse ver­sions. (Au­thors all too often fail to write what they were sup­posed to write—as Yogi Berra re­marked, ⁠“I re­ally didn’t say every­thing I said.”⁠)

• Search Books: Switch to GB and hope some­one para­phrases or quotes it, and in­cludes a real ci­ta­tion; if you can’t see the full pas­sage or the ref­er­ence sec­tion, look up the book in Lib­gen.

Dealing With Paywalls

Gold once out of the earth is no more due unto it; What was un­rea­son­ably com­mit­ted to the ground is rea­son­ably re­sumed from it: Let Mon­u­ments and rich Fab­ricks, not Riches adorn mens ashes. The com­merce of the liv­ing is not to be trans­ferred unto the dead: It is not in­jus­tice to take that which none com­plains to lose, and no man is wronged where no man is pos­ses­sor.

Hy­dri­o­taphia, Urn Bur­ial⁠, Sir Thomas Browne⁠

Use Sci-Hub/Lib­gen for books/pa­persUse uni­ver­sity In­ter­netPub­lic li­braries too

Use Sci-Hub/Lib­gen for books/pa­pers. A pay­wall can usu­ally be by­passed by using Lib­gen (LG)/Sci-Hub (SH): ⁠pa­pers⁠ can be searched di­rectly (ide­ally with the DOI⁠⁠^⁠12⁠, but title+au­thor with no quotes will usu­ally work), or an eas­ier way may be to prepend⁠^⁠13⁠ sci-hub.st (or what­ever SH mir­ror you pre­fer) to the URL of a pay­wall. Oc­ca­sion­ally Sci-Hub will not have a paper or will per­sis­tently error out with some HTTP or proxy error, but search­ing the DOI in Lib­gen di­rectly will work. Fi­nally, there is a Lib­Gen/Sci-Hub full­text search en­gine on the Z-Library mir­ror, which is a use­ful al­ter­na­tive to Google Books (de­spite the poor OCR).

Use uni­ver­sity In­ter­net. If those don’t work and you do not have a uni­ver­sity proxy or alumni ac­cess, many uni­ver­sity li­braries have IP-based ac­cess rules and also open WiFi or Internet-capable com­put­ers with pub­lic lo­gins in­side the li­brary, which can be used, if you are will­ing to take the time to visit a uni­ver­sity in per­son, for using their data­bases (prob­a­bly a good idea to keep a list of needed items be­fore pay­ing a visit).

Pub­lic li­braries too. Pub­lic li­braries often sub­scribe to com­mer­cial news­pa­pers or mag­a­zine data­bases; they are in­con­ve­nient to get to, but you can usu­ally at least check what’s avail­able on their web­site. Pub­lic & school li­braries also have a use­ful trick for get­ting com­mon schooling-related re­sources, such as the OED, or the archives of the New York Times or New Yorker: be­cause of their usu­ally un­so­phis­ti­cated & tran­sient user­base, some pub­lic & school li­braries will post lists of user­names/pass­words on their web­site (some­times as a PDF). They shouldn’t, but they do. Googling phrases like “pub­lic li­brary New Yorker⁠ user­name pass­word” can turn up ex­am­ples of these. Used dis­creetly to fetch an ar­ti­cle or two, it will do them no harm. (This trick works less well with pass­words to any­thing else.)

If that doesn’t work, there is a more opaque ecosys­tem of file­shar­ing ser­vices: booksc/bookfi/bookzz, pri­vate tor­rent track­ers like Bib­li­otik, ⁠IRC⁠ chan­nels with XDCC⁠ bots like #bookz/⁠#ebooks, old P2P net­works like eMule⁠, pri­vate DC++⁠ hubs…

Site-specific notes:

• PubMed: most pa­pers with a PMC⁠ ID can be pur­chased through the Chi­nese scan­ning ser­vice Eu­reka Mag; scans are $38^$30₂₀₂₀ & elec­tronic pa­pers are $25^$20₂₀₂₀.

• El­se­vier/sciencedirect.com: easy, al­ways avail­able via SH/LG

  Note that many El­se­vier jour­nal web­sites do not work with the SH proxy, al­though their sciencedirect.com ver­sion does and/or the paper is al­ready in LG. If you see a link to sciencedirect.com on a pay­wall, try it if SH fails on the jour­nal web­site it­self.

• Psy­c­NET⁠: one of the worst sites; SH/LG never work with the URL method, rarely work with paper ti­tles/DOIs, and with my uni­ver­sity li­brary proxy, loaded pages ‘ex­pire’ and redi­rect while break­ing the browser back but­ton (‽‽‽), com­bined searches don’t usu­ally work (fre­quently fail­ing to pull up even bib­li­o­graphic en­tries), and only DOI or man­ual title searches in the EB­SCO­host data­base have a chance of full­text. (EB­SCO­host it­self is a frag­ile search en­gine which is dif­fi­cult to query re­li­ably in the ab­sence of a DOI.)

  Try to find the paper any­where else be­sides Psy­c­NET!

• Pro­Quest/JSTOR: Pro­Quest/JSTOR are not stan­dard aca­d­e­mic pub­lish­ers, but have ac­cess to or mir­rors of a sur­pris­ingly large num­ber of pub­li­ca­tions.

  I have been sur­prised how often I have hit dead-ends, and then dis­cov­ered a copy sit­ting in Pro­Quest/JSTOR, poorly-indexed by search en­gines.

• Cus­tom jour­nal web­sites: some­times a jour­nal will have its own web­site (eg. Cell or Florida Tax Re­view), but will still be ul­ti­mately run by one of the gi­ants like El­se­vier or HeinOn­line. (You can often see hints of this in the site de­sign, such as the footer, the URL struc­ture, di­rect links to the pub­lisher ver­sion, etc.)

  When this is the case, it is usu­ally a waste of time to try to use the jour­nal web­site: it won’t whitelist uni­ver­sity IPs, SH/LG won’t know how to han­dle it, etc. In­stead, look for the al­ter­na­tive ver­sion.

Request

Human flesh search en­gine

Human flesh search en­gine. Last re­sort: if none of this works, there are a few places on­line you can re­quest a copy (how­ever, they will usu­ally fail if you have ex­hausted all pre­vi­ous av­enues):

• ⁠/r/scholar⁠

• ⁠#icanhazpdf⁠

• Wikipedia Re­source Re­quest⁠

• ⁠LW help desk⁠

Fi­nally, you can al­ways try to con­tact the au­thor. This only oc­ca­sion­ally works for the pa­pers I have the hard­est time with, since they tend to be old ones where the au­thor is dead or un­reach­able—any au­thor pub­lish­ing a paper since 1990_36ya will usu­ally have been dig­i­tized some­where—but it’s easy to try.

Post-Finding

After find­ing a full­text copy, you should find a re­li­able long-term link/place to store it and make it more find­able (re­mem­ber—if it’s not in Google/Google Scholar, it doesn’t exist!):

• Never Link Un­re­li­able Hosts:

  • LG/SH: Al­ways op­er­ate under the as­sump­tion they could be gone to­mor­row. (As my uncle found out with Li­brary.nu shortly after pay­ing for a life­time mem­ber­ship!) There are no guar­an­tees ei­ther one will be around for long under their legal as­saults or the behind-the-scenes dra­mas, and no guar­an­tee that they are being prop­erly mir­rored or will be re­stored else­where.

    When in doubt, make a copy. Disk space is cheaper every day. Down­load any­thing you need and keep a copy of it your­self and, ide­ally, host it pub­licly.

  • NBER: never rely on a papers.nber.org/tmp/ or psycnet.apa.org URL, as they are tem­po­rary. (SSRN is also un­de­sir­able due to mak­ing it in­creas­ingly dif­fi­cult to down­load, but it is at least re­li­able.)

  • Scribd: never link Scribd—they are a scummy web­site which im­pede down­loads, and any­thing on Scribd usu­ally first ap­peared else­where any­way. (In fact, if you run into any­thing vaguely useful-looking which ex­ists only on Scribd, you’ll do hu­man­ity a ser­vice if you copy it else­where just in case.)

  • RG: avoid link­ing to Re­search­Gate⁠ (com­pro­mised by new own­er­ship & PDFs get deleted rou­tinely, ap­par­ently often by au­thors) or Academia.edu (the URLs are one-time and break)

  • high-impact jour­nals: be care­ful link­ing to Na­ture.com or Cell⁠ (if a paper is not ex­plic­itly marked as Open Ac­cess, even if it’s avail­able, it may dis­ap­pear in a few months!⁠^⁠14⁠); sim­i­larly, watch out for wiley.com, tandfonline.com, jstor.org, springer.com, springerlink.com, & mendeley.com, who pull sim­i­lar shenani­gans.

  • ~/: be care­ful link­ing to aca­d­e­mic per­sonal di­rec­to­ries on uni­ver­sity web­sites (often no­tice­able by the Unix⁠ con­ven­tion .edu/~user/ or by di­rec­to­ries sug­ges­tive of ephemeral host­ing, like .edu/cs/course112/readings/foo.pdf); they have short half-lives.

  • ?token=: be­ware any PDF URL with a lot of trail­ing garbage in the URL such as query strings like ?casa_token or ?cookie or ?X (or hosted on S3/AWS); such links may or may not work for other peo­ple but will surely stop work­ing soon. (Acad­e­mia.edu, Na­ture, and El­se­vier are par­tic­u­larly egre­gious of­fend­ers here.)

• PDF Edit­ing: if a scan, it may be worth edit­ing the PDF to crop the edges, thresh­old to bi­na­rize it (which, for a bad grayscale or color scan, can dras­ti­cally re­duce file­size while in­creas­ing read­abil­ity), and OCR it.

  I use gscan2pdf⁠ but there are al­ter­na­tives worth check­ing out.

• Check & Im­prove Meta­data.

  Adding meta­data to pa­pers/books is a good idea be­cause it makes the file find­able in G/GS (if it’s not on­line, does it re­ally exist?) and helps you if you de­cide to use bib­li­o­graphic soft­ware like Zotero⁠ in the fu­ture. Many aca­d­e­mic pub­lish­ers & LG are ter­ri­ble about meta­data, and will not in­clude even title/au­thor/DOI/year.

  PDFs can be eas­ily an­no­tated with meta­data using ExifTool⁠:: exiftool -All prints all meta­data, and the meta­data can be set in­di­vid­u­ally using sim­i­lar fields.

  For pa­pers hid­den in­side vol­umes or other files, you should ex­tract the rel­e­vant page range to cre­ate a sin­gle rel­e­vant file. (For ex­trac­tion of PDF page-ranges, I use pdftk⁠, eg: pdftk 2010-davidson-wellplayed10-videogamesvaluemeaning.pdf cat 180-196 output 2009-fortugno.pdf. Many pub­lish­ers in­sert a spam page as the first page. You can drop that eas­ily with pdftk INPUT.pdf cat 2-end output OUTPUT.pdf, but note that PDFtk may drop all meta­data, so do that be­fore adding any meta­data. To delete pseudo-encryption or ‘pass­worded’ PDFs, do pdftk INPUT.pdf input_pw output OUTPUT.pdf; PDFs using ac­tual en­cryp­tion are trick­ier but ⁠can often be beaten⁠ by off-the-shelf password-cracking util­i­ties.) For con­vert­ing JPG/PNGs to PDF, one can use Im­ageMag­ick for <64 pages (convert *.png foo.pdf) but be­yond that one may need to con­vert them in­di­vid­u­ally & then join the re­sult­ing PDFs (eg. for f in *.png; do convert "$f" "${f%.png}.pdf"; done && pdftk *.pdf cat output foo.pdf or join with pdfunite *.pdf foo.pdf.)

  I try to set at least title/au­thor/DOI/year/sub­ject, and stuff any ad­di­tional top­ics & bib­li­o­graphic in­for­ma­tion into the “Key­words” field. Ex­am­ple of set­ting meta­data:

  exiftool -Author="Frank P. Ramsey" -Date=1930 -Title="On a Problem of Formal Logic" -DOI="10.1112/plms/s2-30.1.264" \
      -Subject="mathematics" -Keywords="Ramsey theory, Ramsey's theorem, combinatorics, mathematical logic, decidability, \
      first-order logic,  Bernays-Schönfinkel-Ramsey class of first-order logic, _Proceedings of the London Mathematical \
      Society_, Volume s2-30, Issue 1, 1930-01-01, pg264–286" 1930-ramsey.pdf

  “PDF Plus” is bet­ter than “PDF”.

  If two ver­sions are pro­vided, the “PDF” one may be in­tended (if there is any real dif­fer­ence) for print­ing and ex­clude fea­tures like hy­per­links .

• Pub­lic Host­ing: if pos­si­ble, host a pub­lic copy; es­pe­cially if it was very dif­fi­cult to find, even if it was use­less, it should be hosted. The life you save may be your own.

• Link On WP/So­cial Media: for bonus points, link it in ap­pro­pri­ate places on Wikipedia or Red­dit or Twit­ter; this makes peo­ple aware of the copy being avail­able, and also su­per­charges vis­i­bil­ity in search en­gines.

• Link Spe­cific Pages: as noted be­fore, you can link a spe­cific page by adding #page=N to the URL. Link­ing the rel­e­vant page is help­ful to read­ers. (I rec­om­mend against doing this if this is done to link an en­tire ar­ti­cle in­side a book, be­cause that ar­ti­cle will still have bad SEO and it will be hard to find; in such cases, it’s bet­ter to crop out the rel­e­vant page range as a stand­alone ar­ti­cle, eg. using pdftk again for pdftk 1900-BOOK.pdf cat 123-456 output 1900-PAPER.pdf.)

Advanced

Aside from the (highly-recommended) use of hotkeys and Booleans for searches, there are a few use­ful tools for the re­searcher, which while ex­pen­sive ini­tially, can pay off in the long-term:

• ⁠archiver-bot⁠: au­to­mat­i­cally archive your web brows­ing and/or links from ar­bi­trary web­sites to fore­stall linkrot; par­tic­u­larly use­ful for de­tect­ing & re­cov­er­ing from dead PDF links

• Sub­scrip­tions like PubMed⁠ & GS search alerts: set up alerts for a spe­cific search query, or for new ci­ta­tions of a spe­cific paper. (Google Alerts⁠ is not as use­ful as it seems.)

  1. PubMed has straight­for­ward con­ver­sion of search queries into alerts: “Cre­ate alert” below the search bar. (Given the vol­ume of PubMed in­dex­ing, I rec­om­mend care­fully tai­lor­ing your search to be as nar­row as pos­si­ble, or else your alerts may over­whelm you.)

  2. To cre­ate generic GS search query alert, sim­ply use the “Cre­ate alert” on the side­bar for any search. To fol­low ci­ta­tions of a key paper, you must: 1. bring up the paper in GS; 2. click on “Cited by X”; 3. then use “Cre­ate alert” on the side­bar.

• GCSE: a Google Cus­tom Search En­gines⁠ is a spe­cial­ized search queries lim­ited to whitelisted pages/do­mains etc. (eg. my Wikipedia-focused anime/manga CSE⁠).

  A GCSE can be thought of as a saved search query on steroids. If you find your­self reg­u­larly in­clud­ing scores of the same do­mains in mul­ti­ple searches search, or con­stantly black­list­ing do­mains with -site: or using many nega­tions to fil­ter out com­mon false pos­i­tives, it may be time to set up a GCSE which does all that by de­fault.

• Clip­pings: note-taking ser­vices⁠ like Ever­note⁠/Mi­crosoft OneNote⁠: reg­u­larly mak­ing and keep­ing ex­cerpts cre­ates a per­son­al­ized search en­gine, in ef­fect.

  This can be vital for re­find­ing old things you read where the search terms are hope­lessly generic or you can’t re­mem­ber an exact quote or ref­er­ence; it is one thing to search a key­word like “autism” in a few score thou­sand clip­pings, and an­other thing to search that in the en­tire In­ter­net! (One can also re­or­ga­nize or edit the notes to add in the key­words one is think­ing of, to help with re­find­ing.) I make heavy use of Ever­note clip­ping and it is key to re­find­ing my ref­er­ences.

• Crawl­ing Web­sites: some­times hav­ing copies of whole web­sites might be use­ful, ei­ther for more flex­i­ble search­ing or for en­sur­ing you have any­thing you might need in the fu­ture. (ex­am­ple: “Dark­net Mar­ket Archives (2013–2015_11ya)”⁠).

  Use­ful tools to know about: wget⁠, cURL⁠, HT­Track⁠; Fire­fox plu­g­ins: ⁠No­Script, uBlock ori­gin⁠, Live HTTP Head­ers⁠, By­pass Pay­walls⁠, cookie ex­port­ing.

  Short of down­load­ing a web­site, it might also be use­ful to pre-emptively archive it by using linkchecker to crawl it, com­pile a list of all ex­ter­nal & in­ter­nal links, and store them for pro­cess­ing by an­other archival pro­gram (see Archiv­ing URLs⁠ for ex­am­ples). In cer­tain rare cir­cum­stances, se­cu­rity tools like nmap⁠ can be use­ful to ex­am­ine a mys­te­ri­ous server in more de­tail: what web server and ser­vices does it run, what else might be on it (some­times in­ter­est­ing things like old anony­mous FTP servers turn up), has a web­site moved be­tween IPs or servers, etc.

Digital

E-books are rarer and harder to get than pa­pers, al­though the sit­u­a­tion has im­proved vastly since the early 2000s. To search for books on­line:

• More Straight­for­ward: book searches tend to be faster and sim­pler than paper searches, and to re­quire less clev­er­ness in search query for­mu­la­tion, per­haps be­cause they are rarer on­line, much larger, and have sim­pler ti­tles, mak­ing it eas­ier for search en­gines.

  Search G, not GS, for books:

  No Books in Google Scholar

  Book full­texts usu­ally don’t show up in GS (for un­known rea­sons). You need to check G when search­ing for books.

  To double-check, you can try a filetype:pdf search; then check LG. Typ­i­cally, if the main title + au­thor doesn’t turn it up, it’s not on­line. (In some cases, the au­thor order is re­versed, or the title:sub­ti­tle are re­versed, and you can find a copy by tweak­ing your search, but these are rare.).

• IA: the In­ter­net Archive has many books scanned which do not ap­pear eas­ily in search re­sults (poor SEO?).

  • If an IA hit pops up in a search, al­ways check it; the OCR may offer hints as to where to find it. If you don’t find any­thing or the pro­vided, try doing an IA site search in G (not the IA built-in search en­gine), eg. book title site:archive.org.

  • DRM workarounds: if it is on IA but the IA ver­sion is DRMed and is only avail­able for “check­out”, you can jail­break it.

    Check the book out for the full pe­riod, 14 days. Down­load the PDF (not EPUB) ver­sion to Adobe Dig­i­tal El­e­ments ver­sion ≤4.0 (which can be run in Wine on Linux), and then im­port it to Cal­i­bre⁠ with ⁠the De-DRM plu­gin, which will pro­duce a DRM-free PDF in­side Cal­i­bre’s li­brary. (Get­ting De-DRM run­ning can be tricky, es­pe­cially under Linux. I wound up hav­ing to edit some of the paths in the Python files to make them work with Wine. It also ap­pears to fail on the most re­cent Google Play ebooks, ~2021.) You can then add meta­data to the PDF & up­load it to LG⁠^⁠17⁠. (LG’s ver­sions of books are usu­ally bet­ter than the IA scans, but if they don’t exist, IA’s is bet­ter than noth­ing.)

• Google Play⁠: use the same PDF DRM as IA, can be bro­ken same way

• HathiTrust⁠ also hosts many book scans, which can be searched for clues or hints or jail­bro­ken.

  HathiTrust blocks whole-book down­loads but it’s easy to down­load each page in a loop and stitch them to­gether, for ex­am­ple:

  for i in {1..151}
  do if [[ ! -s "$i.pdf" ]]; then
      wget "https://babel.hathitrust.org/cgi/imgsrv/download/pdf?id=mdp.39015050609067;orient=0;size=100;seq=$i;attachment=0" \
            -O "$i.pdf"
      sleep 20s
   fi
  done
   
  pdftk *.pdf cat output 1957-super-scientificcareersandvocationaldevelopmenttheory.pdf
   
  exiftool -Title="Scientific Careers and Vocational Development Theory: A review, a critique and some recommendations" \
      -Date=1957 -Author="Donald E. Super, Paul B. Bachrach" -Subject="psychology" \
      -Keywords="Bureau Of Publications (Teachers College Columbia University), LCCCN: 57-12336, National Science Foundation, public domain, \
      https://babel.hathitrust.org/cgi/pt?id=mdp.39015050609067;view=1up;seq=1 https://psycnet.apa.org/record/1959-04098-000" \
      1957-super-scientificcareersandvocationaldevelopmenttheory.pdf

  An­other ex­am­ple of this would be the Well­come Li­brary; while look­ing for An In­ves­ti­ga­tion Into The Re­la­tion Be­tween In­tel­li­gence And In­her­i­tance, Lawrence1931, I came up dry until I checked one of the last search re­sults, a “Well­come Dig­i­tal Li­brary” hit, on the slim off-chance that, like the oc­ca­sional Chi­nese/In­dian li­brary web­site, it just might have full­text. As it hap­pens, it did—good news? Yes, but with a caveat: it pro­vides no way to down­load the book! It pro­vides OCR, meta­data, and in­di­vid­ual page-image down­loads all under CC-BY-NC-SA (so no legal prob­lems), but… not the book. (The OCR is also un­nec­es­sar­ily zipped, so that is why Google ranked the page so low and did not show any re­veal­ing ex­cerpts from the OCR tran­script: be­cause it’s hid­den in an opaque archive to save a few kilo­bytes while de­stroy­ing SEO.) Ex­am­in­ing the down­load URLs for the highest-resolution im­ages, they fol­low an un­for­tu­nate schema:

  1. https://dlcs.io/iiif-img/wellcome/1/5c27d7de-6d55-473c-b3b2-6c74ac7a04c6/full/2212,/0/default.jpg

  2. https://dlcs.io/iiif-img/wellcome/1/d514271c-b290-4ae8-bed7-fd30fb14d59e/full/2212,/0/default.jpg

  3. etc

  In­stead of being se­quen­tially num­bered 1–90 or what­ever, they all live under a unique hash or ID. For­tu­nately, one of the meta­data files, the ‘man­i­fest’ file, pro­vides all of the hashes/IDs (but not the high-quality down­load URLs). Ex­tract­ing the IDs from the man­i­fest can be done with some quick sed & tr string pro­cess­ing, and fed into an­other short wget loop for down­load

  grep -F '@id' manifest\?manifest\=https\:%2F%2Fwellcomelibrary.org%2Fiiif%2Fb18032217%2Fmanifest | \
     sed -e 's/.*imageanno\/\(.*\)/\1/' | grep -E -v '^ .*' | tr -d ',' | tr -d '"' # "
  # bf23642e-e89b-43a0-8736-f5c6c77c03c3
  # 334faf27-3ee1-4a63-92d9-b40d55ab72ad
  # 5c27d7de-6d55-473c-b3b2-6c74ac7a04c6
  # d514271c-b290-4ae8-bed7-fd30fb14d59e
  # f85ef645-ec96-4d5a-be4e-0a781f87b5e2
  # a2e1af25-5576-4101-abee-96bd7c237a4d
  # 6580e767-0d03-40a1-ab8b-e6a37abe849c
  # ca178578-81c9-4829-b912-97c957b668a3
  # 2bd8959d-5540-4f36-82d9-49658f67cff6
  # ...etc
  I=1
  for HASH in $HASHES; do
      wget "https://dlcs.io/iiif-img/wellcome/1/$HASH/full/2212,/0/default.jpg" -O $I.jpg
      I=$((I+1))
  done

  And then the 59MB of JPGs can be cleaned up as usual with gscan2pdf (empty pages deleted, ta­bles ro­tated, cover page cropped, all other pages bi­na­rized), com­pressed/OCRed with ocrmypdf, and meta­data set with exiftool, pro­duc­ing a read­able, down­load­able, highly-search-engine-friendly 1.8MB PDF.

• re­mem­ber the Ana­log Hole⁠ works for pa­pers/books too:

  if you can find a copy to read, but can­not fig­ure out how to down­load it di­rectly be­cause the site uses JS or com­pli­cated cookie au­then­ti­ca­tion or other tricks, you can al­ways ex­ploit the ‘ana­logue hole’—fullscreen the book in high res­o­lu­tion & take screen­shots of every page; then crop, OCR etc. This is te­dious but it works. And if you take screen­shots at suf­fi­ciently high res­o­lu­tion, there will be rel­a­tively lit­tle qual­ity loss. (This works bet­ter for books that are scans than ones born-digital.)

Physical

Ex­pen­sive but fea­si­ble

Ex­pen­sive but fea­si­ble. Books are some­thing of a double-edged sword com­pared to pa­pers/the­ses. On the one hand, books are much more often un­avail­able on­line, and must be bought of­fline, but at least you al­most al­ways can buy used books of­fline with­out much trou­ble (and often for <$13^$10₂₀₁₉ total); on the other hand, while paper/the­ses are often avail­able on­line, when one is not un­avail­able, it’s usu­ally very un­avail­able, and you’re stuck (un­less you have a uni­ver­sity ILL de­part­ment back­ing you up or are will­ing to travel to the few or only uni­ver­si­ties with paper or mi­cro­film copies).

Pur­chas­ing from used book sell­ers:

• Sell­ers:

  • used book search en­gines: Google Books/⁠find-more-books.com: a good start­ing point for seller links; if buy­ing from a mar­ket­place like Abe­Books/Ama­zon/Barnes & Noble, it’s worth search­ing the seller to see if they have their own web­site, which is po­ten­tially much cheaper. They may also have mul­ti­ple edi­tions in stock.

  • bad: eBay & Ama­zon are often bad, due to high-minimum-order+S&H and sell­ers on Ama­zon seem to as­sume Ama­zon buy­ers are eas­ily rooked; but can be use­ful in pro­vid­ing meta­data like page count or ISBN or vari­a­tions on the title

  • good: Abe­Books, ⁠Thrift Books, Bet­ter World Books, B&N, Dis­cover Books.

    Note: on Abe­Books, in­ter­na­tional or­ders can be use­ful (es­pe­cially for be­hav­ioral ge­net­ics or psy­chol­ogy books) but be care­ful of in­ter­na­tional or­ders with your credit card—many debit/credit cards will fail on in­ter­na­tional or­ders and trig­ger a fraud alert, and Pay­Pal⁠ is not ac­cepted.

• Price Alerts: if a book is not avail­able or too ex­pen­sive, set price watches: Abe­Books sup­ports email alerts on stored searches, and Ama­zon can be mon­i­tored via Camel­Camel­Camel (re­mem­ber the CCC price alert you want is on the used third-party cat­e­gory, as new books are more ex­pen­sive, less avail­able, and un­nec­es­sary).

Scan­ning:

• De­struc­tive Vs Non-Destructive: the fun­da­men­tal dilemma of book scan­ning—de­struc­tively de­bind­ing books with a razor or guil­lo­tine cut­ter works much bet­ter & is much less time-consuming than spread­ing them on a flatbed scan­ner to scan one-by-one⁠^⁠18⁠, be­cause it al­lows use of a sheet-fed scan­ner in­stead, which is eas­ily 5x faster and will give higher-quality scans (be­cause the sheets will be flat, scanned edge-to-edge, and much more closely aligned), but does, of course, re­quire ef­fec­tively de­stroy­ing the book.

• Tools:

  • cut­ting: For sim­ple de­bind­ing of a few books a year, an X-acto knife/razor is good (avoid the ‘tri­an­gle’ blades, get curved blades in­tended for large cuts in­stead of de­tail work).

    Once you start doing more than one a month, it’s time to up­grade to a guil­lo­tine blade paper cut­ter (a fancier swinging-arm paper cut­ter, which uses a two-joint sys­tem to clamp down and cut uni­formly).

    A guil­lo­tine blade can cut chunks of 200 pages eas­ily with­out much slip­page, so for books with more pages, I use both: an X-acto to cut along the spine and turn it into sev­eral 200-page chunks for the guil­lo­tine cut­ter.

  • scan­ning: at some point, it may make sense to switch to a scan­ning ser­vice like ⁠1Dol­larScan (1DS has ac­cept­able qual­ity for the black-white scans I have used them for thus far, but watch out for their nickel-and-diming fees for OCR or “set­ting the PDF title”; these can be done in no time your­self using gscan2pdf/exiftool/ocrmypdf and will save a lot of money as they, amaz­ingly, bill by 100-page units). Books can be sent di­rectly to 1DS, re­duc­ing lo­gis­ti­cal has­sles.

• Clean Up: after scan­ning, crop/thresh­old/OCR/add meta­data

  • Adding meta­data: same prin­ci­ples as pa­pers. While more elab­o­rate meta­data can be added, like book­marks, I have not ex­per­i­mented with those yet.

• File for­mat: PDF, ⁠not DjVu⁠

  De­spite being a worse for­mat in many re­spects, I now rec­om­mend PDF and have stopped using DjVu for new scans⁠^⁠19⁠ and have con­verted my old DjVu files to PDF.

• Up­load­ing: to Lib­Gen, usu­ally, and Gwern.net some­times. For back­ups, file­lock­ers like Drop­box, Mega, Me­di­aFire, or Google Drive are good if you have no web­site of your own. I usu­ally up­load 3 copies in­clud­ing LG. I ro­tate ac­counts once a year, to avoid putting too many files into a sin­gle ac­count. [I dis­cour­age re­liance on IA links.⁠)

  • Do Not Use Google Docs/Scribd/Drop­box/IA/etc for Long-Term Doc­u­ments

    ‘Doc­u­ment’ web­sites like Google Docs (GD) should be strictly avoided as pri­mary host­ing. GD does not ap­pear in G/GS, doom­ing a doc­u­ment to ob­scu­rity, and Scribd is lu­di­crously user-hostile with chang­ing dark pat­terns⁠. Such sites can­not be searched, scraped, down­loaded re­li­ably, clipped, used on many de­vices, archived⁠^⁠20⁠, or counted on for the long haul. (For ex­am­ple, Google Docs has made many doc­u­ments ‘pri­vate’, break­ing pub­lic links, to the sur­prise of even the au­thors when I con­tact them about it, for un­clear rea­sons.)

    Such sites may be use­ful for col­lab­o­ra­tion or sur­veys, but should be re­garded as strictly tem­po­rary work­ing files, and moved to clean sta­tic HTML/PDF/XLSX hosted else­where as soon as pos­si­ble.

• Host­ing: host­ing pa­pers is easy but books come with risk:

  Books can be dan­ger­ous; in de­cid­ing whether to host a book, my rule of thumb is host only books pre-2000 and which do not have Kin­dle edi­tions or other signs of ac­tive ex­ploita­tion and is ef­fec­tively an ‘or­phan work⁠’.

  As of 2019-10-23, host­ing 4,090 files over 9 years (very roughly, as­sum­ing lin­ear growth, <6.7 mil­lion document-days of host­ing: 3,763 × 0.5 × 8 × 365.25 = 6,722,426), I’ve re­ceived 4 take­down or­ders: a be­hav­ioral ge­net­ics text­book (2013_13ya), The Hand­book of Psy­chopa­thy (2005_21ya), a re­cent meta-analysis⁠ paper (Roberts et al 2016), and a CUP DMCA take­down order for 27 files. I broke my rule of thumb to host the 2 books (my mis­take), which leaves only the 1 paper, which I think was a fluke. So, as long as one avoids rel­a­tively re­cent books, the risk should be min­i­mal.