Cory Doctorow: Zuck’s Empire of Oily Rags

For 20 years, privacy advocates have been sounding the alarm about commercial online surveillance, the way that companies gather deep dossiers on us to help marketers target us with ads. This pitch fell flat: by and large, people were skeptical of the efficacy of targeted advertising; the ads we got were rarely very persuasive, and when they did work, it was usually because the advertisers had figured out what we wanted and offered to sell it to us: people who’d previously shopped for a sofa saw ads for sofas, and if they bought a sofa, the ads persisted for a while because the ad targeting systems weren’t smart enough to know that their services were no longer needed, but really, where was the harm in that? The worst case scenario was that advertisers would waste their money with ads that had no effect, and the best case scenario was that shopping would get somewhat more convenient as predictive algorithms made it easier for us to find the thing we were just about to look for.

Privacy advocates tried to explain that persuasion was just the tip of the iceberg. Commercial databases were juicy targets for spies and identity thieves, to say nothing of blackmail for people whose data-trails revealed socially risky sexual practices, religious beliefs, or political views.

Now we’re living through the techlash, and finally people are coming back to the privacy advocates, saying we were right all along; given enough surveillance, companies can sell us anything: Brexit, Trump, ethnic cleansing in Myanmar, and successful election bids for absolute bastards like Turkey’s Erdogan and Hungary’s Orban.

It’s great that the privacy-matters message is finally reaching a wider audience, and it’s exciting to think that we’re approaching a tipping point for indifference to privacy and surveillance.

But while the acknowledgment of the problem of Big Tech is most welcome, I am worried that the diagnosis is wrong.

The problem is that we’re confusing automated persuasion with automated targeting. Laughable lies about Brexit, Mexican rapists, and creeping Sharia law didn’t convince otherwise sensible people that up was down and the sky was green.

Rather, the sophisticated targeting systems available through Facebook, Google, Twitter, and other Big Tech ad platforms made it easy to find the racist, xenophobic, fearful, angry people who wanted to believe that foreigners were destroying their country while being bankrolled by George Soros.

Remember that elections are generally knife-edge affairs, even for politicians who’ve held their seats for decades with slim margins: 60% of the vote is an excellent win. Remember, too, that the winner in most races is “none of the above,” with huge numbers of voters sitting out the election. If even a small number of these non-voters can be motivated to show up at the polls, safe seats can be made contestable. In a tight race, having a cheap way to reach all the latent Klansmen in a district and quietly inform them that Donald J. Trump is their man is a game-changer.

Cambridge Analytica are like stage mentalists: they’re doing something labor-intensive and pretending that it’s something supernatural. A stage mentalist will train for years to learn to quickly memorize a deck of cards and then claim that they can name your card thanks to their psychic powers. You never see the unglamorous, unimpressive memorization practice. Cambridge Analytica uses Facebook to find racist jerks and tell them to vote for Trump and then they claim that they’ve discovered a mystical way to get otherwise sensible people to vote for maniacs.

This isn’t to say that persuasion is impossible. Automated disinformation campaigns can flood the channel with contradictory, seemingly plausible accounts for the current state of affairs, making it hard for a casual observer to make sense of events. Long-term repetition of a consistent narrative, even a manifestly unhinged one, can create doubt and find adherents – think of climate change denial, or George Soros conspiracies, or the anti-vaccine movement.

These are long, slow processes, though, that make tiny changes in public opinion over the course of years, and they work best when there are other conditions that support them – for example, fascist, xenophobic, and nativist movements that are the handmaidens of austerity and privation. When you don’t have enough for a long time, you’re ripe for messages blaming your neighbors for having deprived you of your fair share.

But we don’t need commercial surveillance to create angry mobs: Goebbels and Mao did it very well with analog techniques.

Facebook isn’t a mind-control ray. It’s a tool for finding people who possess uncommon, hard-to-locate traits, whether that’s “person thinking of buying a new refrigerator,” “person with the same rare disease as you,” or “person who might participate in a genocidal pogrom,” and then pitching them on a nice side-by-side or some tiki torches, while showing them social proof of the desirability of their course of action, in the form of other people (or bots) that are doing the same thing, so they feel like they’re part of a crowd.

Even if mind-control rays remain science fiction, Facebook and other commercial surveillance platforms are still worrisome, and not just because they allow people with extreme views to find each other. Gathering huge dossiers on everyone in the world is scary in and of itself: in Cambodia, the autocratic government uses Facebook to identify dissidents and subject them to arrest and torture; the US Customs and Border Protection service is using social media to find visitors to the US guilty by association, blocking them from entering the country based on their friends, affiliations and interests. Then there are the identity thieves, blackmailers, and con artists who use credit bureau data, leaked user data, and social media to ruin peoples’ lives. Finally, there are the hackers who supercharge their “social engineering” attacks by harvesting leaked personal information in order to effect convincing impersonations that trick their targets into revealing information that lets them break into sensitive networks.

It’s fashionable to treat the dysfunctions of social media as the result of the naivete of early technologists, who failed to foresee these outcomes. The truth is that the ability to build Facebook-like services is relatively common. What was rare was the moral recklessness necessary to go through with it.

The thing is, it’s always been obvious that by spying on internet users, you could improve the efficacy of advertising. That’s not so much because spying gives you fantastic insights into new ways to convince people to buy products as it is a tribute to just how ineffective marketing is. When an ad’s expected rate of success is well below one percent, doubling or tripling its efficacy still leaves you with a sub-one-percent conversion rate.

But it was also obvious from the start that amassing huge dossiers on everyone who used the internet could create real problems for all of society that would dwarf the minute gains these dossiers would realize for advertisers.

It’s as though Mark Zuckerberg woke up one morning and realized that the oily rags he’d been accumulating in his garage could be refined for an extremely low-grade, low-value crude oil. No one would pay very much for this oil, but there were a lot of oily rags, and provided no one asked him to pay for the inevitable horrific fires that would result from filling the world’s garages with oily rags, he could turn a tidy profit.

A decade later, everything is on fire and we’re trying to tell Zuck and his friends that they’re going to need to pay for the damage and install the kinds of fire-suppression gear that anyone storing oily rags should have invested in from the beginning, and the commercial surveillance industry is absolutely unwilling to contemplate anything of the sort.

That’s because dossiers on billions of people hold the power to wreak almost unimaginable harm, and yet, each dossier brings in just a few dollars a year. For commercial surveillance to be cost effective, it has to socialize all the risks associated with mass surveillance and privatize all the gains.

There’s an old-fashioned word for this: corruption. In corrupt systems, a few bad actors cost everyone else billions in order to bring in millions – the savings a factory can realize from dumping pollution in the water supply are much smaller than the costs we all bear from being poisoned by effluent. But the costs are widely diffused while the gains are tightly concentrated, so the beneficiaries of corruption can always outspend their victims to stay clear.

Facebook doesn’t have a mind-control problem, it has a corruption problem. Cambridge Analytica didn’t convince decent people to become racists; they convinced racists to become voters.

This review and more like it in the July 2018 issue of Locus.

While you are here, please take a moment to support Locus with a one-time or recurring donation. We rely on reader donations to keep the magazine and site going, and would like to keep the site paywall free, but WE NEED YOUR FINANCIAL SUPPORT to continue quality coverage of the science fiction and fantasy field.