Recent searches
Search options
Administered by:
More fun publisher surveillance:
Elsevier embeds a hash in the PDF metadata that is *unique for each time a PDF is downloaded*, this is a diff between metadata from two of the same paper. Combined with access timestamps, they can uniquely identify the source of any shared PDFs.
You can see for yourself using exiftool.
To remove all of the top-level metadata, you can use exiftool and qpdf:
exiftool -all:all= <path.pdf> -o <output1.pdf>
qpdf --linearize <output1.pdf> <output2.pdf>
To remove *all* metadata, you can use dangerzone or mat2
Also present in the metadata are NISO tags for document status indicating the "final published version" (VoR), and limits on what domains it should be present on. Elsevier scans for PDFs with this metadata, so good idea to strip it any time you're sharing a copy.
Links:
exiftool: https://www.exiftool.org/
qpdf: https://qpdf.sourceforge.io/
dangerzone (GUI, render PDF as images, then re-OCR everything): https://dangerzone.rocks/
mat2 (render PDF as images, don't OCR): https://0xacab.org/jvoisin/mat2
here's a shell script that recursively removes metadata from pdfs in a provided (or current) directory as described above. For mac/*nix-like computers, and you need to have qpdf and exiftool installed:
https://gist.github.com/sneakers-the-rat/172e8679b824a3871decd262ed3f59c6
![[Screenshot of code at URL in tweet, the script first uses "find" to locate the files, and passes them to a while loop. It creates a clean PDF at a temporary file, removing it if one exists already. Code follows]
# Color Codes so that warnings/errors stick out
GREEN="\e[32m"
RED="\e[31m"
CLEAR="\e[0m"
# loop through all PDFs in first argument ($1),
# or use '.' (this directory) if not given
DIR="${1:-.}"
echo "Cleaning PDFs in directory $DIR"
# use find to locate files, pip to while read to get the
# whole line instead of space delimited
# Note -- this will find pdfs recursively!!
find $DIR -type f -name "*.pdf" | while read -r i
do
# output file as original filename with suffix _clean.pdf
TMP=${i%.*}_clean.pdf
# remove the temporary file if it already exists
if [ -f "$TMP" ]; then
rm "$TMP";
fi
exiftool -q -q -all:all= "$i" -o "$TMP"
qpdf --linearize --replace-input "$TMP"
echo -e $(printf "${GREEN}Processed ${RED}${i} ${CLEAR}as ${GREEN}${TMP}${CLEAR}"](https://social-coop-media.ams3.cdn.digitaloceanspaces.com/media_attachments/files/107/686/442/772/750/424/original/f5b43f49b4762cd1.jpg "[Screenshot of code at URL in tweet, the script first uses \"find\" to locate the files, and passes them to a while loop. It creates a clean PDF at a temporary file, removing it if one exists already. Code follows]
# Color Codes so that warnings/errors stick out
GREEN=\"\e[32m\"
RED=\"\e[31m\"
CLEAR=\"\e[0m\"
# loop through all PDFs in first argument ($1),
# or use '.' (this directory) if not given
DIR=\"${1:-.}\"
echo \"Cleaning PDFs in directory $DIR\"
# use find to locate files, pip to while read to get the
# whole line instead of space delimited
# Note -- this will find pdfs recursively!!
find $DIR -type f -name \"*.pdf\" | while read -r i
do
# output file as original filename with suffix _clean.pdf
TMP=${i%.*}_clean.pdf
# remove the temporary file if it already exists
if [ -f \"$TMP\" ]; then
rm \"$TMP\";
fi
exiftool -q -q -all:all= \"$i\" -o \"$TMP\"
qpdf --linearize --replace-input \"$TMP\"
echo -e $(printf \"${GREEN}Processed ${RED}${i} ${CLEAR}as ${GREEN}${TMP}${CLEAR}\"")
The metadata appears to be preserved on papers from sci-hub. since it works by using harvested academic credentials to download papers, this would allow publishers to identify which accounts need to be closed/secured
https://twitter.com/json_dirs/status/1486135162505072641?t=Wg5XAzujycz79Cop_ap8vQ&s=19
for any security researchers out there, here are a few more "hashes" that a few have noted do not appear to be random and might be decodable. exiftool apparently squashed the whitespace so there is a bit more structure to them than in the OP:
https://gist.github.com/sneakers-the-rat/6d158eb4c8836880cf03191cb5419c8f
this is the way to get the correct tags:
(on mac i needed to install gnu grep with homebrew `brew install grep` and then use `ggrep` )
will follow up with dataset tomorrow.
https://twitter.com/horsemankukka/status/1486268962119761924?s=20
of course there's smarter watermarking, the metadata is notable because you could scan billions of pdfs fast. this comment on HN got me thinking about this PDF /OpenAction I couldn't make sense of earlier, on open, access metadata, so something with sizes and layout...
![[top comment on HN thread]
So just take pics of the pages and convert the pics back to a PDF
[first sub-comment]
A motivated publisher could embed codes by altering in subtle ways the differences in distances or color between adjacent characters, so that they would survive most color or grey scale conversions; a seemingly innocuous frame drawn around a photo could be either larger or smaller by say one millimeter, representing de facto a bit, therefore using enough pages they could identify a book among billions. Unfortunately there's no way to be 100% sure that a complex document doesn't contain some form of embedded code.
[second sub-comment]
Easier to just strip out the metadata](https://social-coop-media.ams3.cdn.digitaloceanspaces.com/media_attachments/files/107/688/129/541/792/457/original/8eb65aca095b33e0.jpg "[top comment on HN thread]
So just take pics of the pages and convert the pics back to a PDF
[first sub-comment]
A motivated publisher could embed codes by altering in subtle ways the differences in distances or color between adjacent characters, so that they would survive most color or grey scale conversions; a seemingly innocuous frame drawn around a photo could be either larger or smaller by say one millimeter, representing de facto a bit, therefore using enough pages they could identify a book among billions. Unfortunately there's no way to be 100% sure that a complex document doesn't contain some form of embedded code.
[second sub-comment]
Easier to just strip out the metadata")
updated the above gist with correctly extracted tags, and included python code to extract your own, feel free to add them in the comments. since we don't know what they contain yet not adding other metadata. definitely patterned, not a hash, but idk yet.
https://twitter.com/json_dirs/status/1486289288115359747?t=QwmBvbOgh2fCkjSOZSh3Fw&s=19
you go to school to study "the brain" and then the next thing you know you're learning how to debug surveillance in PDF rendering to understand how publishers have so contorted the practice of science for profit. how can there be "normal science" when this is normal?
follow-up: there does not appear to be any further watermarking: taking two files with different identifying tags, stripping metadata, and relinearizing with qpdf's --deterministic-id flag yields PDFs identical with a diff, ie. no differentiating watermark (but plz check my work)
which is surprising to me, so I'm a little hesitant to make that as a general claim
@jonny they look kind of meaningful. Not base64. Any ideas what could be in there?
@derwinmcgeary
yeah, I thought so too but don't know where to start reverse engineering it :/
@derwinmcgeary
it decodes with base85, but it's not Unicode. not sure if that's meaningful
@jonny I do not have any IT skills, but if I did I’d love to write a script to remove metadata from PDFs. Adobe has them wrapped up pretty well.
@beckett @jonny "PDFparanoia" was a project for exactly this - to strip identifying watermarks and metadata from shared academic PDFs. But it fell victim to the Python 2 to 3 transition and the mess of the PDF libraries in particular, and then fell to bitrot. Would be nice to see it brought back to health.
@seachaint
@beckett
yes, lives on in mat2 and I think pdfparanoia specifically redirects to dangerzone
@matthew
pretty straightforward to get at least the top level metadata
https://social.coop/@jonny/107686442819944047
The cognitive load of constantly dealing with greedy corporate rentiers is exhausting.
It's a couple things:
a) Elsevier's vendor's tool only has to be good enough to impress Elsevier
b) Deterrence being more efficient than prevention
@jonny Some other publishers render a unique PDF that has the download date, user, and institution. It is not in the metadata but a visible watermark on the edge of the pages.
@EricFielding @jonny I have that in a stats book I bought from Springer
@jonny for the normativity of science see the discourse of STS (science and technology studien), great field!
@shusha
yes definitely, love it and spend basically all my time reading it nowadays 
@jonny I wonder whether uploading every paper to sci-hub twice would be feasible (i.e. would we still have enough people do that). (If we did so, then it would allow sci-hub to verify with reasonable certainty that whatever watermark-removal method they would use still works.)
@robryk
I think it may be easier to scrub it server side, like to have admins clean the PDFs they have. I don't know of any crowdsourced sci-hub-like projects. scrubbing metadata does seem to render the PDFs identical
@jonny And then obviously the watermarking techniques will adapt. Asking for two copies is a way to ensure that whatever we are doing still manages to scrub the watermark (they should be identical after scrubbing).
@robryk
yes, definitely. all of the above. fix what you have now, adapt to changes, making double grabs part of the protocol makes sense :)
@jonny hmmm, alternatively start inserting copies of that metadata into blank or template PDFs. Send 'em chasing wild geese and make them look at Lorem Ipsum 50 times a day 
@http_error_418 @jonny Very well.
Do you have code to insert those code ? I would like to help.
Btw, you can use mathgen https://thatsmathematics.com/mathgen/ to make meaningless paper to upload on scribd
@jonny Aren't these the same guys who charges you (or your institution) a sh*tload of money to publish these works in the first place?
fuchsiaaaaaaaaaaaaaaaaa @jonny word of caution is that while removing exif is good, knowing publishers there's a bunch of other ways they'd directly include such trackers into the file, in a less human/machine readable spot than EXIF. so be careful
@jonny (1/2) Why remove metadata if you could overwrite crucial parts with some similarly looking crap?
We use computationally expensive hashes and ciphers not to punish users for mistyped passwords with delaying next attempt, but because it transfers the brute forcing costs to attackers.
Don't make it immediately obvious to bad actors that their methods have been figured out
@jonny (2/2) Plant mindfucks - garbage hashes meeting original format, randomized dates (reasonable timeframe - not earlier that potential earliest records, not earlier timestamp than given document version etc), and any kind of poisoning that will make the (meta)data unobviously worthless.
Use ethical methods to fuck with unlawful actors. You won't beat the bigger guy using the same weapon.
@wojtek
Be my guest!
@jonny Well, this is the first time I read the name "Elsevier" and I can't ask you to throw any copyright-protected files my way. Treat it as a hint to improve your script :-)
@jonny they're almost getting to the level of ISO standards for metadata f'wittery.
For a while, many ISO standards that you bought (for $$$$) looked like a bad photocopy. If you zoomed in really close to the marks on the page, they were made up of a pattern of punctuation characters. Totally screwed up any screen reading, though
@jonny Adding unique identifiers on stuff you distribute to be able to trace where it gets copied is hardly a new thing, and I don't think it is good terminology to call it "surveillance". As the hash is a passive part of the document, it is not used (and possibly can't be used) to spy on you.
I don't think it is productive to call this practice "surveillance", as it just make it more difficult for the readers to differentiate between levels of threats to their privacy.
@jonny@social.coop seems like some countermeasures against scihub, libgen and other shadow libraries that provide those PDFs for free 
@jonny this is the same technique that was being used in the OS designed in North Korea called Red Star OS. It was in the Chaos Congress talk about it.
@kawaiipunk
interesting ... will take a look.
@jonny they really are a right bunch of bastards, aren’t they?
@jonny this makes me think some horrible things are going to happen to students because of this but I can’t quickly think of an example
@jonny But what meaningful data can they attach to that unique ID? The IP adress? Assume a user is not logged in, has cleared tracking cookies and is using a VPN.
Wait a sec. That is why we need open access.
@cmkobel
Browser fingerprinting is p robust.
https://www.amiunique.org/fingerprint
And even those mitigations wont be taken by 99.999% of visitors.
@jonny You know how AAAS and T&F have some "Preparing your PDF" loading screen when you click "Download article"? I'm pretty that's what's happening, on the fly.
You don't need those loading pages for static assets, like PDFs. You would want loading screens like that, though, to smooth over the creation of a unique PDF with identifiable metadata in it every time someone clicks that button.
Good advice on how to view and/or remove the hash