Some really good thoughts here. I'll summarize the ones that hit me:
- "Why are people searching Reddit specifically? The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust."
This is it for me exactly. I search for the following kinds of things on Reddit exactly because results on other sites aren't trustworthy: Reviews are secretly paid ads. The "best" recipe for pancakes is only what's trending on instagram right now. The latest conditions on mountain bike and hiking trails are being shared inside communities like Reddit but not on the web. The same for trending programmer tools.
- "It is obvious that serving ads creates misaligned incentives for search engines..."
What I'm shocked by is that Google somehow maintained a balance on this for so long. Well, at least a good enough balance that people still use it primarily.
- "Google increasingly does not give you the results for what you typed in. It tries to be “smart” and figure out what you “really meant" ..."
This is the most annoying behavior because I really mean what I write.
- "There’s a fun conspiracy theory that popped up recently called the Dead Internet Theory..."
I hadn't heard of this. Now that's some sci-fi level of conspiracy but in today's world it seems totally plausible.
There's a reason why it seems shocking that Google has been able to balance the ads well enough that people still use it. They haven't! Google has orchestrated a monopoly over search engine distribution that allows them to get away with search results that are dominated by ads and spam, without losing most consumers.
Let's be blunt here - almost no consumer consciously chooses to use Google search anymore. Google has a distribution monopoly through Android, its deal with Apple on iOS and MacOS, and on desktop through Chrome.
I'm working on a search engine startup. It is in all practical senses impossible for an iPhone or Mac user to change their search engine to a new search engine on Safari or at the iOS level. And despite being technically possible on desktop with Chrome, it is for all practical purposes beyond what any typical consumer can easily do.
Their monopoly over distribution - not search result quality - is what keeps consumers searching Google and clicking ads.
> It is in all practical senses impossible for an iPhone or Mac user to change their search engine to a new search engine on Safari or at the iOS level.
On my IOS device, under Settings -> Safari -> Search Engine, I have a drop down with options, including Bing and DuckDuckgo, but defaulted to google.
On Macos, with Safari running, Safari -> Preferences… -> Search, Search Engine I have a drop down, defaulted to google, with Bing and DuckDuckgo amongst other choices.
Agreed on google”s effort to get their search engine as the default. However I just don’t understand how changing search engine is impossible given what I’m seeing on my devices? Nor does it seem over the top onerous to my eyes.
Yes, but you can't add a new search engine at all! So if a search engine isn't one of the tiny number of options in that dropdown, you can't change to it. That applies on both iOS and MacOS. And that option is used for the entire system-wide search, not just Safari.
So here's a challenge, try adding a search engine not on that list. You can see the search engine I'm working on in my profile if you're interested (I don't want to hijack this thread with self-promotion). I challenge you to change to a new competitive option like it. You simply can't. That is a clear monopoly over distribution.
On desktop in Chrome, as noted it is not something any typical consumer can do easily. But even if they could, Google does not allow you to set the New Tab to another search engine, even by setting the homepage to one. So every new tab opened on Chrome takes you back to Google search, even if a consumer figures out how to change their homepage. As for changing the nav bar search, no ordinary consumer is going to be able to work out how to change a search URL pattern. That is clearly intended to prevent consumers changing.
So I stand by my point, especially on an iPhone, you simply cannot change your search engine to a new search engine like us. It is impossible.
I fully understand your point and defaults are very strong.
That being said, I try new search engines from time to time and always get back to google, because non of the others have worked for me (in a professional context). I probably do 200 searches per day and google is most likely to give me relevant info on my first query (maybe 80-90% success rate). All others I have tried have been around 40-50% win an avg. of 2-3 search queries to find my result. That is a huge daily time sync on 200 searches per day.
I will also test your search engine.
And before having tested it, I have some unsolicited advice ;) At least these are things that would make me switch:
1) you are strong in my vertical. 50% of my daily search queries are professional. Probably 10-20% are programming related. If you were better 20% better than google at delivering results for that subset, I would probably use you.
2) If you had very strong support for my locale. Based in Germany, 50% of my private searches are in German. Most search engines, apart from google, suck in German. My assumption is their market share is so small that they don‘t put effort into any language specific search syntax understanding. German or large language groups like Spanish, Hindi, French come to mind.
3) If you can‘t become a default search engine on safari, maybe you can role your own browser (chromium fork or something) where you are the default. You could package it as: MySearchEngine App. It is actually a fully functional browser, but users really use it because they want to use your search engine. That might give easier access than having to manually navigate to your website in safari.
The ultimate test of any search engine is always the results. While the project I'm working on is definitely still an alpha, I'd love to chat with you when you try it out. I don't want to take the conversation here off-topic. To your general points though, there are definitely opportunities to provide better results within specialist areas of knowledge, and for local markets.
I think most of the search startups are doing their own mobile app. On iOS, the system search and browser remains Google/Safari (and the App is essentially just a wrapper on Safari for browsing). But at least it is something. I think you'll see more people doing desktop apps, although the dominance of Google through Chromium forks for this isn't a coincidence. It feels like the bad old days of re-packaging Internet Explorer with a custom homepage all over again.
Multi-lingual search results in general; Google (and FB) just for some reason cannot comprehend that someone might have reasons to search in multiple languages or even regions. For instance, I at one point was an editor/fact-checker for an academic publisher and one project involved checking a lot of information from official government sources around the world, and Google did not know what to do with that.
Hell, I'd love to have the option to make the search results be completely language agnostic.
Don't you just set the region and language at the bottom?
I also search in multiple languages but I also know that's an exception. The majority of the people in the world likely would only use a single language and it's easy to believe that search results are better for those users with languages separated.
Yes, hence why I'd want an option for language agnostic results. It would be a terrible default.
There are cases when I'd even like REGION agnostic results, or least ones not bound by national boundaries. For example, I have an interest in my state history (MI), and there's plenty of relevant articles/commentary from the Canadians.
I don't think it's too niche, I think they're just American.
A Canadian/Indian/EU-based Google probably would've ended up with multi-lingual support, whereas a China/UK based one probably wouldn't have.
By the time Google got big enough for international considerations, there were already a bunch of baked in assumptions about their users, like them being monolingual.
Metager.org is my go to google search replacement. I imagine (although my German is non native) that the German search results are quite good. . . it being a German search engine. To be honest it the only search engine to get me off google - DDG is OK, as is searx, but I kept going back to the dirty G until Metager.
Also a fan of Brave browser here. The more people trying new things the better! I know some people don't like the crypto-angle with it, but I think it is a positive to test new economic models to support online content, personally.
At least in part this seems to be due to the profiles that Google has built. Whenever I try to use Google in anonymous mode, the results become noticeably worse.
Which, of course, means that it is yet another barrier to entry for any would-be competitors.
This could also be a perverse incentive reflected algorithmically (whether on purpose or not). Google has better data and makes more money when users are identified. So they have a vested interest in making users think that they will get worse results in Incognito mode.
Personalization is a double-edged sword. @pg once wrote about Google results becoming "what's true for you" rather than what's objectively true. And filter bubbles and subtle "personalization censorship" are also dangers. I think it's possible to have high quality results and privacy/anonymity, and it's not a binary choice. It's a challenge worth figuring out.
How much of the German challenge do you think is due to German being an inflected language? So a search engine that can't figure out the inflected variants of a word are going to potentially miss out on a large number of the relevant search results.
In addition to inflections, another factor may be the tendency to use compound words, so that a relevant word in context might not have a space at each end.
In English, Google is pretty good at figuring out when to add spaces in your queries (if you have a typo with your spaces). Similar technology should help it with German?
There's nothing magic about putting a space between words or not. Similar to whether you hyphenate or not.
Maybe it isn't a big deal; my thought was that if you're crawling a website and trying to associate its contents with search terms, parsing an uncommon compound word into the correct parts is going to come up more often in German than in English, especially if you're smaller than Google and German isn't your main focus. Are search engines good enough now to know that a string like electricovenmonitor is about ovens but not covens or Venmo?
Thanks, just to let you know, yes, it does have an opensearch description. But in practical terms that doesn't help much even if a startup search engine adds it.
Unfortunately, while OpenSearch is great where it's supported, outside of Firefox, the only real support is for in-site-search on other platforms (where you type a site name and then a search string), and not for changing your browser search engine. And it doesn't work at all on iOS even for site search.
So unfortunately it doesn't fix the problem of how a consumer can easily change their search engine to something new on Chrome or Safari or an iPhone.
I don't want to sidetrack the discussion, but if you want to confirm the opensearch description, you can open our site in Firefox, then click the "..." in the browser address bar and then click "Install Andi Search". Or reach out and very happy to talk you through it.
Depending on the platform, you might need to right-click on the Firefox browser address bar and choose the "Add search engine" option from the dropdown (where a website has it). Do you see an option to add a new search engine when you right-click the browser address bar at all?
[Edit: just saw the edit that you found it - thanks again for looking around for it too!]
I concur. And would add that on Safari and iOS, it suits Google and Microsoft too keep others out; noting all options are Google, Bing or Bing sydnicates). And it suits Apple nicely; $15 bn from Google, pure margin. How much do they get from Microsft/syndicates? Meanwhile all search listed options in Chrome are Google or Microsoft (Bing and Bing syndicates). And, to complete all Edge options are Bing, Bing syndicates or Google. Disclosure: also alternative search engine CEO.
Presumably you count DuckDuckGo as a "Bing syndicate". I think that doesn't do it justice - many of the things I like about DDG are specific to DDG, and I could not care less whether the underlying crawl was run by Bing or not.
I see your point. I would have sworn that DuckDuckGo was added as a search entry when I installed that app on my ios device, however my memory is hazy from that long ago, so perhaps that search engine was added at a different point, like when it became big enough for Apple to notice them.
I'm a huge fan of DuckDuckGo. My understanding is that it took a significant amount of effort and public lobbying for them to get added to that list, and it was back in 2014 that was announced.
I suspect the big problem is that even if your search engine succeeds, once you get enough traction you'll just sell up to Google and everything slots back to normal 'do some evil' mode.
I don't want to get too off-topic, but personally I can promise you we will never sell to Google. This problem is very personal. As well as being a programmer, I used to work as a journalist, and I've seen friends in media have their lives and businesses destroyed by Google's ad-tech. And I've seen the Internet turn from a wonderful thing to become a cesspool of content farms, clickbait and seo spam. I can't speak for other search startups, but we will never run ads or use ad-tech or sell user data or invade people's privacy. We're just two people. But we're two people on a mission.
We pay nothing so are not included. On the other hand we are happy to provide a one click search from our search engine or eight other search engines/services, currently from our web app [0]. One click for Google (if that's your thing), one click for Bing or Gigablast search engine results, one click for Brave and some of the many Bing and few Google syndicates - DuckDuckGo, Ecosia, Startpage.
Even if DuckDuckGo and Bing pay absolutely nothing, it's good leverage to make Google pay even more, and its already public knowledge Google pays a lot to remain the default search engine on iOS, to the tune of billions of dollars a year:
You have a section in Andi Search which lets you try and set it as the default (in Firefox at least) but it doesn't work. Is that what you mean, or am I doing something wrong/weird?
BTW I really like what you're doing, and I'll definitely like to set it as default in Firefox somehow if I could just to try it out. My first tests were surprisingly good, I didn't expect it to deliver results which are accurate and which appear to at least match DDG. Nice one. [slightly unusual page format on desktop though :)]
Thanks! I don't want to distract from the main topic too much but would love to chat with you if you're open to it. There's a thread below with @neffity and I made some changes (trying to get Chrome on iOS to use OpenSearch) and broke that for a little. Update is just deploying now. I sincerely appreciate the feedback and you having a look too! :)
Firefox supports an open search standard which is a big improvement, so that's probably where you saw this. It provides an easier experience from the nav bar to add a new search engine to the browser. In practice, while it's a huge improvement, I've found talking with users that it's mostly helpful if someone technical is talking them through the change.
Hello Jed, that sucks. I understand that custom search engine exclusion is a financial decision by companies that can get away with almost whatever they want to do.
For iPhone, iPad, and macOS, you could use Swift and SwiftUI to write a single search app for your service. Flutter is pretty good also, then you could cover Windows and Linux also.
That's simply not true. Even as a technical user who knows what you're doing, you get a hard-coded choice of exactly five - Google, Bing, Yahoo, DDG or Ecosia. You can't add new ones at all. If you wanted to add a new search engine (like the one I'm working on, or any other), there is no way to do it.
But let's try it. Go to Safari > Preferences > Search and choose the "Search engine" dropdown, and you'll see you have a dropdown with only five options.
Google reportedly pays $15B per year for that top spot.
A work around is to add a safari extension like "Keyword Search" and associate a key word to your search. Its nothing like setting a default search engine for a normal user, but atleast there is a way for the slightly more technical user.
Though, I couldn't add a keyword to your site in firefox with its "Add a keyword" feature. Might be due to the javascript calling some url behind the screen.
Adding a shortcut to a website to the home screen is a nice convenience. Unfortunately, it doesn't add it as a search engine to your iPhone search, or to Safari. Even with a PWA (progressive web app), the App is just a wrapper on Safari. For a startup like us, it is probably the best option currently available, and we've tried to make it a good experience. But it does suffer the same drawbacks as a regular App as a browser replacement. If you swipe down from the home screen and type something into your iPhone's search bar, you're still using Google and Safari.
The OpenSearch standard is great and definitely an improvement where it's supported, but unfortunately as far as I'm aware in most browsers that's limited to site-search (in website search after typing a url). It doesn't work at all on iOS.
On Firefox it makes it easier to talk a non-technical user through how to change their default search engine, and at least they aren't entering search pattern strings into a settings field as with Chrome.
As far as I'm aware and the OpenSearch docs, support is limited to in-site searches on most platforms outside of Firefox (not changing default search option for example using Chrome on iOS). Would be really interested in the steps you followed if you're happy to share them.
I then went to my site on Chrome iOS and it showed up under Settings > Search Engine > Recently visited. I then selected it and now anything I put in my url bar gets sent to my server. It's pretty sick.
That's interesting, I don't get that option on Chrome on iPhone (Search Engine > Recently visited), except for the officially supported five (Google, Bing, Yahoo, DDG, Ecosia).
I'm wondering do you see that as an option visiting our site (we have an opensearch description for Andi also) or other search engines like You with one?
Thanks for having a look. The opensearch file is https://andisearch.com/andi.xml, and it's correctly referenced from the "<link rel="search"..." opensearch tag. If you have a look in Firefox you'll see it prompts in the nav bar. So I don't think that would be the issue. Do you see anything for other engines like You with the opensearch head item?
I'm thinking maybe Chrome on iOS has its own flavor of implementation and possibly it needs the filename to be "opensearch.xml" rather than looking at the <link> tag. I'm renaming the file to see if that makes a difference. If you had an example of another site where it worked, that would be awesome. At least one more platform where OpenSearch works for setting the default site search would be nice.
I'd be interested in the steps you took. Third-party Apps on iOS can't change the Safari or system-wide options for search engines as far as I'm aware. Installing a third-party App just gives you a wrapper around Safari for browsing while you use that App only. If you swipe down on your home screen to use the system search, or open Safari itself, nothing has changed. I can ask you to install our App or another search provider's App, but it doesn't change your iPhone's search engine or add it to Safari.
Yes, I found it a few weeks ago. It changes how safari works and improves it quite a lot.
I can block elements, and cookies in a site, define custom JavaScript or css, change the default search engine with one in a list o define a new one, and a whole lot of other things. All inside default safari.
That's interesting. I didn't think an extension could change the search engines available under Settings > Safari > Search Engine on iOS from the defaults (Google, Bing, Yahoo, DDG, Ecosia), or change the system-wide search used. Do you see different options now under Settings for search engines and did it change your system-wide search too (i.e. swipe down on home screen and then use the built-in Search)?
> especially on an iPhone, you simply cannot change your search engine to a new search engine like us. It is impossible.
See my post above about iCabMobile, where there are TWENTY-FIVE search engines to choose from.
There are also other browsers than Safari and iCabMobile on iOS, many of which give alternatives to their search engine choices.
Naturally if you think only users who choose the default browser are interesting as your market, I wonder if those users would take a chance on your alternative search engine?
I replied to your duplicate post separately, but it is worth noting that even if you install an alternative browsing App and use it, the iOS system search (swipe down from the top to access the search bar) is still using Google and Safari. And even if you were to use another browser like iCabMobile, it also simply does not let you add in a new search engine not already in its own options.
> And even if you were to use another browser like iCabMobile, it also simply does not let you add in a new search engine not already in its own options
I duplicated my post because you so pervasively in this thread duplicated your inaccuracies.
You are wrong about this, iCabMobile allows the user to very easily add search engines:
Settings > Tools > Search Engines > Add
Naturally I expect to be plentifully downvoted for this accurate information, just like pointing-out that iCabMobile has 25 search engines to was also extremely unpopular.
I may not be following you, but are you saying that when you change search engine in your App, it also changes the default search for iOS and Safari system-wide?
Or you are just talking about within your iCabMobile App itself, like anyone can do in their own app?
As far as I know, all the alternative browser and search Apps (including ours for example) are wrappers on top of Safari on iOS. And they are unable to change the iPhone's system-wide search or browser. That's the search engine accessed, for example, when a user swipes down on their iPhone home screen, and enters a query into the "Search" field that appears. Normally, that's Google and Safari at the iOS system-wide level.
Or are you saying that you are able to change these including adding alternative search engines under iOS Settings > Safari > Search Engine?
I'm curious how you're doing this if so, and I'm sure lots of other people would be too. My understanding is that App Store Apps or Extensions can not normally change the iPhone search engines, but I'd love to be wrong about that.
> Google does not allow you to set the New Tab to another search engine
Firefox has removed the ability to set a default page for new tabs and requires users to install an extension to restore the functionality, which in fact provides degraded functionality. As originally implemented the new tab would load the new page instantly. With the extension, a new tab is created, focus is given to the URL bar and after a brief but noticeable pause, the chosen page loads.
xSearch for Safari makes the search experience on the Apple ecosystem much better. https://apps.apple.com/us/app/xsearch-for-safari/id157990206...
Makes switching search engines easy and you can use "bangs" like in DuckDuckGo to use other search engines within safari.
Apple is operating a search engine right under our noses, hiding behind Siri. Applebot is regularly crawling the web. They’re probably much further along toward a privacy-focused search product than any of us would know.
"Web results" are. "Siri Suggested Websites" and "Siri Knowledge" are not — those are fed by Applebot. "Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot." — Apple's support document about Applebot.
I've wondered why no one has bought wolfram alpha and used it to power knowledge bases. Certainly it seems cheaper for Apple to have down that as a start point than to start from scratch. Maybe it's too technical for most users?
"Let's be blunt here - almost no consumer consciously chooses to use Google search anymore."
This bluntness does not go far enough. People do not change defaults, no matter how "easy" it may be to do so.
A default is a pre-made choice by someone other than the consumer. There is no set-up process where the consumer makes a choice. The choice has already been made. Consumers do not make this choice. Even if they could, in practice they don't. That fact may seem insignificant but it is worth billions of dollars.
If I am not mistaken, the current CEO of Google spent most of his time working on "default search engine" (or "default web browser") deals before taking the CEO job. In probably the most important one, Google pays Apple a hefty sum to be the default search engine. It was estimated at $10 billion in 2020 and $15 billion in 2021.[1]
Defaults are effectively permanent settings. It does not matter how easy it is to change a default setting if practically no one ever does it. $15 billion is too much to pay for something that may or may not change. It does not change. It is money in the bank.
> If I am not mistaken, the current CEO of Google spent most of his time working on "default search engine" (or "default web browser") deals before taking the CEO job.
I mean you could spend two seconds to search and realize you were in fact mistaken before bothering to write "If I am not mistaken..."
Sundar Pichai was responsible for the Chrome browser and Android operating system. [1]
While the comment might be a little oversimplified, I think it's reasonable to say that those deals would have fallen under his ambit. And there's no question that Chrome and Android are the two central planks of Google's search distribution monopoly with consumers, along with the Apple deal.
> While the comment might be a little oversimplified
let's see:
> spent most of his time working on "default search engine" (or "default web browser") deals before taking the CEO job
even if we take your convenient assumption as correct ("it's reasonable to say that those deals would have fallen under his ambit"), it's still wouldn't be a true statement. But feel free to point out which part is true if we relax any more assumptions.
Based on the public record, I don't think the original commenter's statement was unreasonable or mistaken. And they have provided extensive additional documentation on a separate comment supporting the assertion.
Based on Pichai being the senior executive at Google responsible for Chrome browser and search defaults, as a matter of public record he held corporate responsibility for getting google search as the default search engine on as many devices as possible.
You stated that the original commenter was mistaken with no supporting evidence and a high level of acrimony in your phrasing. I'd be interested to see supporting evidence for why you think the commenter was mistaken, in the context of the other resources they provided.
Sure, if you're comfortable with your job being described as spending most of your time working on affiliate link deals, we can agree that's a good description of someone working on the Google toolbar plugin and then Chrome OS, Android, etc.
If anyone else was curious, Sundar Pichai had not worked on Search prior to becoming CEO, it seems:
> Pichai joined Google in 2004,[8] where he led the product management and innovation efforts for a suite of Google's client software products, including Google Chrome and Chrome OS, as well as being largely responsible for Google Drive. In addition, he went on to oversee the development of other applications such as Gmail and Google Maps. In 2010, Pichai also announced the open-sourcing of the new video codec VP8 by Google and introduced the new video format, WebM. The Chromebook was released in 2012. In 2013, Pichai added Android to the list of Google products that he oversaw.
More about Pichai's role in making Google the default search on more computers and directing search traffic to Google, by any means necessary.
"Pichai started at Google leading product management for the Google toolbar, a critically strategic product that enabled default search queries on different web browsers to go through Google and allow them to track browsing behavior to power the AdWords targeting engine. At the time, Internet Explorer was the "installed by default" incumbent for many users, while Firefox was the alternative browser of choice."
"Pichai identified a weakness in Google's strategy, and Chrome began as a defensive play against the established browsers to protect and grow Google's search business (which still generates much of the company's revenue)."
"Sundar Pichai is the one who introduced the toolbar, which led to an increase in user searches. It was later merged with Chrome, which became the most used web browser in the world."
""Most people here didn't want us to do a browser, so it was a little bit stealthy. Once we had it up and running, I remember showing it to Larry and Sergey - and even then there was a lot of scepticism." But Pichai got his way: Chrome was released in 2008 and now accounts for nearly 60% of the market, according to NetMarketShare, while Internet Explorer languishes on less than 16%."
"Ten years ago, the Indian-born Pichai, 42, was a product manager at Google, and his domain consisted of the search bar in the upper right corner of Web browsers. He then persuaded his bosses to wade into the browser wars with Chrome, which in time became the most popular browser on the Internet and led to the Chrome operating system that runs on a line of cheap laptops called Chromebooks."
"Android runs on 1.2 billion devices around the world. It drives users to the company's hugely profitable search engine and the ads on its maps service. Google search and maps are available on phones made by Apple and Microsoft, too, but Google pays those companies referral fees. The more people use Android, the more Google can keep that revenue to itself."
"Google distributes the latest versions free under the agreement that device makers will highlight profitable Google services-especially search and maps-while their own brands and services take a back seat."
"Pichai joined the small team working on Google's search toolbar. It gave users of Internet Explorer and Firefox, the dominant browsers at the time, easy access to Google search. He proposed that Google build its own browser and won the support of the company's co-founders, though he faced an objection from then-CEO Eric Schmidt, who thought that joining the browser wars would be an expensive distraction."
"Rubin had introduced it, but Pichai created an interdisciplinary team with the search group, which had voice search technology and algorithms that could discern what information might be most important to users. "Sundar helped me to formalize a relationship," says Johanna Wright, the product manager who runs Google Now. "Because search and Android sit in two different buildings, we ended up doing a people swap.""
Thank you for putting that together. I knew there was some resistance to doing a browser internally, but it was a brilliant insight that long term Google would need control of distribution for its search results as it became more and more a mass consumer service, especially after becoming ads-focused and moving toward more mainstream users, and with the world moving to mobile after the iPhone launched.
Consumers don’t want to see the results of their search or find the answer to their question. They want the assurance of being told the answer by an authority. Google has tried to become that authority. It’s true, just Google it.
Just tried https://andisearch.com/ and I like it. Felt like a fresh look on results instead of the same old SEO ones. For example, searched for a few Java queries and found very informative website/results that weren't dominated by Bealdung. Searched for "soccer scores", "chelsea FC", "prince andrew", "WP export" and found things that would never have been on Google's first page, but were excellent returns. Nice work.
Thank you. While I don't want to distract from the topic here, I do really appreciate your feedback and you trying it out, and would love to talk more if you'd like to reach out! Speaking generally, the world needs more people working on search, and I think there is a lot of room for completely new approaches.
I'm pleasantly surprised by the high quality of your top results. That shows the search algorithms (whatever your choice was) are good enough and Google can be eventually disrupted. As a fellow software engineer I'm curious about your stack and size of your operations. Are you blogging about this somewhere? You definitely should!
> That shows the search algorithms ... are good enough and Google can be eventually disrupted
I doubt this. I'm guessing that 90% of Google Search's development effort is concentrated on defeating SEO, and as soon as a new competitor succeeds well enough to attract SEO attention, their results will suddenly turn to custard.
The competitor then has to combat the extremely sophisticated SEO practises developed against Google over the years. This is likely to be an even bigger barrier to disruption than building market share against Google.
You're assuming here that Google's primary interest is consumers, but their paying customers are the advertisers. The reality is that the worse the organic results are, the more likely a user is to click on the ads instead, and that's how Google makes money.
At the least that's a perverse incentive, and at worst it's a corrupting influence.
Whether it is conscious or not, Google does better financially when there is more SEO spam in results.
That's why the better Google does financially, the worse the search results are getting.
I doubt this very much myself too! I little wishful thinking to be honest.
However you're arguing as if new competitors had to monetize the same way Google does and that's not the only game available. Imagine a competitor that doesn't monetize clicks but works on donations (like wikipedia does). If results are higher quality than Google's the users will follow. Or maybe a search engine that promotes a particular product like a CRM tool for example.
Disruption typically doesn't come in obvious ways or else someone else would have done it (including Google and its gazillion dollars).
Thank you. I appreciate that very much. We plan to share on HN how we've built it and how it works, and get as much feedback as we can. That really needs its own thread, and I'm conscious of not getting off-topic or promotional here because this is such an interesting thread already, and it wasn't my intention to distract from that because this is one of the big problems in the world right now. Plus, to be honest, we have a few things that are broken and that we're working to fix! So to answer your question, yes! We hear the feedback asking us to share more, and we're going to get a new release up and share with HN soon. And sincerely, thanks for the encouragement to do that. We think it will be interesting to the community here.
Hey thank you. Seriously I appreciate the feedback and encouragement to share Andi with HN. I know there have been a few comments and questions, even though I'm trying to not get off-topic because this is such an interesting thread. You're absolutely right that this needs it's own thread where we can answer questions properly.
So we're definitely planning to share what we're building with the community here soon. Andi's very much an alpha and a few things are broken, but we're working on a new release that fixes them, and then we're going to share it and learn as much as we can from the feedback and wisdom here.
use the "go" command (e.g. go reddit)
close the newly opened tab (e.g. reddit)
click "Learn more about me"
click the top left icon (Andi)
=> goes directly to reddit.com
Thank you! And ugh! I thought we fixed that bug. On it :)
One of the secrets with Google is search is that the top searches aren't searches at all. Number one search on google? "facebook" - it's people navigating. So the idea behind that is to let people navigate directly when they want to go somewhere specific, like "go youtube cute cats".
Without wanting to go off topic here, may I ask what sort of error did you get? All looks fine from here, although it is an alpha. If you were happy to reach out I can try to help figure it out with you.
Oh awesome, thank you! Yes just say "/bug" any time and tick the little box to attach details (we don't see what searches are so that let's us know) or just hop on our Discord from Contact Us. We really appreciate you trying it out. It is an alpha and we have lots of work to do, especially performance. I've been trying not to go off topic from the conversation here but wanted to answer your direct question, but we can chat more directly.
I love the HN view option...mostly because I find that form so easy to scan.
I've conducted a number of professional searches about neuroscience, obscure R packages, topics on D&D and Pathfinder, and it has done exceptionally well. The one thing my obscure demographic would love is a replacement for Google Scholar, which I use all the time.
I love the HN view too - it is compact, efficient and information dense. I think it works well for search results. And I love HN so it seemed like a neat homage :)
Thank you for trying it out and the great feedback too. There hasn't been too much discussion here about Google Scholar, but better research and academic results is something I hear a lot of people talk about as a growing problem.
Secure Connection Failed: An error occurred during a connection to andisearch.com. PR_CONNECT_RESET_ERROR. Also Firefox is flagging it as "Not Secure" in the address bar.
Interestingly, once I turned off my VPN, it works again, Firefox doesn't display the message either. Note that other web pages work fine on the VPN.
update: It work's with MozillaVPN, but not with my University's VPN. Perhaps UC Davis has flagged it from within their firewall.
Very cool effort but just the first thing I tried which is a mega common search for everyone. "Denver weather."
Took a while to load, and only gave me short text of current conditions.
When I did "denver weather forecast" it gave a bit more and showed results on the right from Wolfram and others but they don't tell me anything. Wolfram is current conditions and wind tomorrow. Next one just meta description for weather underground.
It's going to snow tomorrow. That's what people are searching for.
Google gets this stuff and makes that info super visually accessible.
I really don't get the hate on their search quality. If it has gone down I haven't experienced this outside of mega spam things like recipes.
But I still experience the black magic of typing in some vague thing you are trying to remember and somehow they know what it is.
I've been trying not to be off-topic or self-promotional on this thread because it has so much other great discussion, and it needs a separate thread, but I wanted to answer your question quickly. Thanks sincerely for the feedback and for trying it out. We'd really love to chat more with you about this if you're open to it. The project is still an alpha and we have lots of work to do. We outline some of the good and bad things on our About page, including that it's weak at local searches (like weather or businesses or localizing to region) and there is too much spam in product reviews and ecommerce especially.
My own feeling is that the spam and ad problem (and therefore actual result quality) on Google is at its worst for categories like finance, health and travel. Commercial promotion has really taken over there. For many people, ads are essentially spam when they take over the entire screen of results (try, say, "best home loans in the US").
There is huge variation though. Some searches are still pure. Thanks again and please feel welcome to reach out to chat more about this!
There's a reason why it seems shocking that Google has been able to balance the ads well enough that people still use it. They haven't! Google has orchestrated a monopoly over search engine distribution that allows them to get away with search results that are dominated by ads and spam, without losing most consumers.
I disagree. Two to three years ago I could get more what I wanted in a complex search once I tuned it properly. So Google had a twenty year run of good and useful searches. Google also worked to strong arm their monopoly, yes. But I claim they still served some quality after that. It's not that unusual for a monopoly built on quality to maintain their quality for a period of time after it achieves that monopoly status - institutional standards die but they can die over time.
Is there a reason why you've chosen the chat style interface vs the standard search box at the top and results at the bottom layout?
This is not a comment on the search results itself - always appreciate the efforts to break out of the standard google results and surface other sources, but I found the interface confusing and the previews were also taking up a lot of space. A compact view would be better - or giving the option to turn the previews on / off.
Thanks for trying it out. I don't want to take to discussion too off topic, but if I can try to answer in general terms, I think that it is not just that the search results on Google have been getting worse, the user experience has been too. There has been very little real innovation in how search works in the last 20 years, so it is good to try new approaches.
With different views like compact vs visual, our feeling is that it's good to give people choices. If you get chance on desktop with Andi, try a search and then under Search Results, click "Change View", and try some of the other views. List view gives straight compact text results, and there is a Hacker view that presents results in the same information dense view as HN. That's my favorite. There is even a view Goggles that has a similar format to Google circa 2000 :)
I'd love to chat more with you about this, and it is a great topic for when we share what we're building in its own thread here. Just based on feedback, we have two fairly passionate groups of early users on this topic - some love the conversational interface and others just want it to look like Google. So the approach we're taking is to give people choice.
> almost no consumer consciously chooses to use Google search anymore...Their monopoly over distribution - not search result quality - is what keeps consumers searching Google
I don't disagree with this as a fact, but I think there are a lot of things that work this way that aren't actually monopolies in the competition-preventing sense. If I wanted to launch a new breakfast cereal, getting my product into grocery stores would be one of the major challenges of starting that business. Competition for shelf space is a core concern of a lot of consumables. This definitely creates a lot of stickiness and barriers, and that comes with its share of downsides, but there are also good reasons that distribution systems work the way they do. Transaction costs are important.
I don't think competition for shelf space is the right analogy here. Perhaps for Apps within the App Store you could argue that. But when there are only two mobile operating systems with meaningful market share, and when they make it impossible to change to a new search engine at all, and the results all come from only two sources (Google or Bing) that's a straight monopoly over distribution.
It's a similar situation with the App Stores also. They are monopolies. We've gone from a world of personal computing where software was a free market with open choices, to a closed and proprietary world where there is only one available source of software.
> We've gone from a world of personal computing where software was a free market with open choices, to a closed and proprietary world where there is only one available source of software.
That’s true but at the same time I think most people are pretty happy with it. HN readers aren’t typical in this regard.
I’ve been writing software as my job since the mid-80’s and it’s only been in the past 4 or 5 years where I realized that I’m finally pretty happy with the tech I use day-to-day.
If I had any complaint it would be that app stores have made software too inexpensive. When I look at something like Procreate which I think cost something like $10, I’m blown away. This can’t be sustainable.
You have a point but shelf space is physically limited. Online real estate is not so limited. In my country there is reasonably healthy competition between supermarkets. Supermarkets do have self-branded products but they don't cross-sell competitors self-branded products.
Here we have Apple with Google and Bing on their shelves. Microsoft have Bing and Google on their shelves. And Google have Goggle or Bing. Is that healthy or an oligopoly?
There's a reason web designers call specific pages "valuable real estate".
For example Google's search page, the one with the input, is probably the most valuable web real estate in the world, closely followed by the first page of results once you've typed your query and hit Enter.
I'm willing to bet $100 that the second page of results probably gets less than 1000th the hits the first one gets. Heck, make that 1 millionth of the hits the first results page gets.
That's silly. Everything is limited by the scarcity of human attention spans, not just websites.
Shelf space refers to the market with which someone competes, not whether people are thinking of a candy bar or finding a bathroom or a facebook post. Your argument commits survivor bias because it's ignoring the millions of other websites that exist and are being used. Being popular does not mean something is a monopoly, nor does it mean there's limited shelf space.
Following your example, if Google spammed Pixel ads on it's home page, the page would become less popular. One of the reasons it became so popular was it's strict adherence to focusing on utility.
Even if true, what does that have to do with shelf space? Search results do not constitute the internet, and there are many more search engines than just Google.
And then there's the problem of the difference between your cereal and the big ones aren't going to be big because cereal is a finished art. The same with search. My outcomes using bing or google are almost the same. The reality is a lot of good conversation is locked within social media discussions and reddit is the only one that allows it all to be public by default, hence google + reddit. We're moving to walled gardens and most of them will simply keep google out. Google is probably as good as it can be, but it just doesn't have access to discussions in places like private facebook groups, discord, etc.
Not to mention reddit is terrible outside of tech concerns. It leans conservative, young, male, and white. As a woman, contributing there is an invitation for harassment. Even when I don't contribute I do things like research cars to buy only to see forums dominated by "car guys" who mock safety standards, focus only on performance and the "get laid" aspect of cars, and are dismissive towards groups like "soccer moms." Well, I'm a soccer mom and felt wholly unwelcome in those communities and the advice there is actually terrible advice for parents wanting to buy cars.
Then there's a whole religion on gaming be it consoles vs consoles or vs pcs, or publishers or franchises and just endless tribalism. Politics is an absolute nightmare as you can imagine. The savvy reader will say "well you have to know how to sort the comments a special way and never visit the sub you want but the 'true' version of the sub you want, etc" which is a million times more hostile to non-technical people than scrolling past some google ads and finding an article about what they want from a reputable newpaper or consumer reports.
Reddit is just too wild west to be a google replacement. Worse, the good content is almost impossible to find. Google weighs popular discussions more than recent ones so it keeps giving me discussions from many years ago, even if I try to put the year in the search box. It has no idea how to crawl reddit and make it digestible for us and the reddit leadership want nothing to do with google it seems, for capitalistic reasons of keeping out a potential competitor and having their search be really good "any day now."
Posting to reddit is its own kind of nightmare, full of rules per sub, each different and with an algo that will decide if your question gets any visibility, often only getting mean spirited comments in return, if not harassment. For as far as facebook has fallen, I can still visit my town or neighborhood group and ask people what cars they like and get something of a normal discussion. At reddit, I'll be asked for nudes or just mocked/gatekept.
If anything if google is an old man at web 1.0 then reddit is an old man at web 2.0. I suspect google with outlast reddit as reddit looks primed to be overtaken, a bit like how myspace, slashdot and digg looked like unstoppable juggernauts during their time. Its "manboy" culture and its super hostile default subs and everyday misogyny, transphobia, and racism scare normal people away. Forums have always been the seedy underbelly of the internet but reddit is seemingly proud of being seedy, with spez coming out to say that he welcomes covid disinformation when reddit was recently called out about how its become the home of covid conspiracy theories.
If reddit was publicly traded and you asked me between google and reddit, I'd say buy google 100%. I think we'll be reading a lot more "what happened to reddit" articles in the near future, not "what happened to google" articles. Reddit becomes more toxic over time and that's just advertiser poison. Remember, it happily was the home of "jailbait" subs showing sexualized photos of children and "creepshots" showing non-consensual photos of random women until CNN called them out. If there was no call out, then spez would happily be selling those subs as points of pride and growth. "Reddit is the new google" narratives are very echo-chambery, shortsighted, and highly questionable. I think it forever remains this cesspool that chases away advertisers while Google continues to adapt to a new online world and continues to be vastly profitable. Meanwhile, Reddit has yet to make a profit.
I've changed my default search engine to Bing for a while. Before I did that, I did compare the results with Google search and found that the clickbait websites that keep pissing me off are shown only by Google search. Those content farm sites have been on Google search as the top results for years to the extent I think it literally cancels any advantage Google search provides.
Every time I hear this "Reddit is conservative-biased" I wonder if people are living in the same reality as I am. I just checked /r/all, the first political post is related to the Canadian trucker protest of which support is broadly split down the political divide. Top voted posts there slide as I expected, generally on the left-aligned component against the protests. The same, at least in my experience, applies to all political topics on reddit.
Hell, Bernie, and then Biden were both top of reddit during the elections. In what way is that indicative of a "conservative leaning" on reddit. I'll give you young and male, but conservative? You've got to be pulling my leg here.
/r/all is just a spam of new items, not a view of its culture.
It was the largest Donald Trump fan site in the world (/r/thedonald before it was shut down for being so big its brigading tactics were damaging the site) and now /r/conservative takes that role. Before it was the biggest Ron Paul fan site. Its deeply pro-guns in nearly every comment section and any mention of Hillary or AOC is an invitation for angry comments and downvotes. Random subs are full of transphobic content and misogyny is near everywhere. /r/conspiracy is a right-wing paradise catering to right wing views.
You cannot talk about money or finance without being yelled at about how bad fiat and the fed are, which are right-wing talking points. You can't have a covid discussion without an army of covid skeptics yelling at everyone. You can't discuss police brutality without dishonest "but both sides" types full of racist dog whistles.
Its absolutely right leaning and token "liberal" positions like legal pot or better healthcare doesn't really change that. I know GOP voters with those positions but they always vote GOP for culture-war related reasons. Not to mention, being able to do drugs in your home is really neither liberal or conservative, its just in the USA only the liberal party is making any effort to make that happen.
Covid skepticism, which spez defends, is extremely coded right-wing, so even leadership acknowledges who reddit users really are.
The few liberal and feminist spaces that exist on reddit have draconian mod rules because of the constant brigading and some of them just give up and lock discussions because mods are exhausted and tired of fighting it. Just running a liberal or feminist or queer sub is a lot of work because reddit conservatives are constantly bridaging. This is not the sign of a liberal community. I mod a few subs and its just a nightmare out there. Just keeping conservatives from trolling and fighting with everyone is a big job. If you see "liberalism" its because the mods are keeping the everyday redditors away and trying to keep up the values and themes of that specific sub and its relatively tiny audience compared to the conservative majority.
Cherry picking AI spammed Bernie posts doesn't change the culture in the comments or what visitors receive when they post. I'm in a lot of Democratic, liberal, feminist, queer, socialist, etc spaces and I can guarantee you reddit is absolutely nothing like those spaces.
Also you are taking my comment out of context, not only is reddit conservative, but as a hypothetic competitor to google its extremely conservative. I can google for things without being hit with Ron/Rand Paul narratives, Trump worship, pro-gun narratives, covid skepticism, transphobia, or racist dog whistles. So just the idea that Reddit is as welcoming as a Google search is very, very questionable. Google will transparently present you ads. Reddit will drown you in propaganda, culture war politics, and harassment. As the meme says, we are not the same.
My suspicion is that any broadly successful community aggregator will have enough people holding belligerent, incompatible views, that one could never feel truly safe participating there.
Your loathing (for lack of a better term) of the discourse on Reddit has me wondering if the differences I perceive between our political and cultural tolerance stems from being on opposite sides of a few gnostic / agnostic boundaries, or if the difference between our demographic pigeon holes has spared me orders of magnitude of relative grief, allowing me the additional advantage of getting less overwhelmed by ambient asshattery.
I will readily admit that the latter could easily lead to the former; I guess I wonder if the 2 —> 1 causal order holds overwhelmingly in practice, and how to satisfactorily determine that in a way that manages to be honest, systematic, and compassionate.
>My suspicion is that any broadly successful community aggregator will have enough people holding belligerent, incompatible views, that one could never feel truly safe participating there.
I'm of the opinion that this is fine. Rather, the better approach here might be instead gatekeeping and exclusivity. Now, before you tear my head off for that, let me elaborate a bit.
Much of the issues of existing communities, seem to me to stem from an inability to deal with scale. A single heretic isn't a problem, but when the heretics outnumber the believers, they can then proceed to dominate the community. One potential solution I've wondered about is entry-restrictions and finer grained restrictions on permissions for a community. For instance, a community might be public view but member-only for posts/comments, with invite-only membership. Or perhaps memberships have to be approved by N randomly selected members.
The catch of course, is that what I propose is also yet another contributor to the death of the open internet, much like discord is doing.
Reddit as a whole is absolutely a community and the default subs and the comments in them reflect that. Its also one that brigades a lot into other subs, so there's a "real reddit" you see and its absolutely right-leaning.
/r/all sorted by hot is items roughly sorted by popularity. Only if you sort by new, would it be a flood of contents by time.
We can take a quick survey of /r/all sorted by "hot" as of this moment. The very first post I see with AoC is literally this:
"""AOC tells Joe Biden: Cancel more student loans to have "any chance" in 2022"""
The top posts? All in support. Where is the dominant conservative force that warps conversation around it.
HermanCainAward, literally a subreddit celebrating the deaths of the unvaccinated tops all regularly. How the hell is that "covid skeptic right dominant"?
Is your idea of "reddit is conservative" that reddit hosts conservative communities at all? Even if they're smaller than the left-aligned communities there?
EDIT:
Some other quick statistics:
Size of /r/conservative: 923k
Size of /r/lgbt: 864k
Size of /r/hermancainaward: 500k
The "lol antivaxxers died" community is literally half the size of the largest conservative subreddit. The LGBT subreddit is the same size as OP's bugbear /r/conservative. The very much Dem supporting /r/politics is an order of magnitude larger than either of those, at 8 MILLION subscribers.
Yeah, I was waiting for this comment. The OP you're responding to is way way WAY overexagerating the opinions of the body politic of reddit.
Reddit is so large that having a potent conservative force is an inevitability - and much as we don't want to admit, for all the nastyness that goes on there, it's still a space that forces conservatives to be relatively speaking on "good behavior" (imagine if gab/voat/parler successfully displaced reddit), and more recently also forces their ideology to be heavily moderated (as in, to become more moderate), and watered down.
Also LOL at criticism of the car community for being bad at recommending SUVs. Reddits car enthusiast community is made up of people whose favorite car by and large is the Miata. Is the Miata a good car for a soccer mom? No. Is the Miata the car of dudebro conservatives? Uh, HELL NO!
I hate to say it, but sorry Karen, Reddit is for your son, not you.
“…almost no consumer consciously chooses to use Google search anymore”? 6-8 quality leads in the last 14 days (ave. sale at $3,200) on less than $220 spent on ads begs to differ with you. We’ve only started advertising the last two weeks. We’ve had calls and form submissions _all_ from Google and we only launched our site roughly 45 days ago. I’m not a Google fanboy and I think Search does need an overhaul but people are mostly definitely using Google Search. Another client of mine gets 8-12 new customers per month all from Google searches and she doesn’t spend a dime on Google.
In many ways I think that supports my comment. People use Google search because it's the default and a monopoly, and it has a total monopoly on search ads as a result. But that is not a choice that consumers consciously make to go and use it. It's pre-installed as the only easily accessible default on their phones and computers, and no one ever thinks about what search engine they use or has chosen to use it. If you buy an Apple or Android phone, your search engine is Google, and you just assume it is the only search engine. It's great that it gets good results for your ad spend for you. That's why Google continues to set new records for revenue. Advertisers like you are their true customers. And the people searching are the product being sold.
They were not saying "nobody uses Google search" but rather that people were not consciously choosing to use it over other services, but using it because it is the default on virtually every device and browser, despite the fact that the majority of the results were ads. The fact that people are clicking on your ads doesn't exactly disprove that hypothesis.
"almost no consumer consciously chooses to use Google search anymore"
May I ask how you arrived at this observation? This is the first time I am hearing this. I know of NO ONE who uses any other search engine. The term "Googled" is not yet a proxy for other search sites.
People use Google because it's what comes installed on their phones and computers when they first turn them on, and they never actively choose it. So while everyone uses it, few consumers make an active choice to use it. From talking to users a lot, many just assume it's the only option - as you say, Google has become synonymous with search in the same way Xerox became a word for photocopying.
Consumers use Google because it's the default and the only visible option when they turn their phones on. Unfortunately, that's what a monopoly on distribution looks like. People no longer make a choice and don't even realize they have one.
I would think that there'd be an online opportunity for a search engine that only searches humanly curated sites. Those sites would be ones that have quality information rather than spam. Some obvious examples - wikipedia, reddit, hackernews, public domain books, etc.
It's easy to game an algorithm, but hard to game a human - humans know garbage when they see it.
As an aside, whenever I get a prescription, included with it is a dense two page sheet of detailed information about the drug. I see nothing like that online with a search. Why is this sort of thing not online?
At least France and Belgium have public websites with the information sheets of all authorized drugs. I think at least the French one generally comes up in the first results on Google (when searching from a French connection).
pretty much what we're building at breezethat.com -- currently launching about one topic / week, and opening door soon so others can curate / moderate a topic
Maybe Yahoo's time has come again! Maybe Google's decline started when they no longer had competition from Yahoo?
The interesting thing would be coming up with a sustainable business model for it. One way might be the users pay for it, either per-search or per-month. This way the incentives to provide good search results align with the interests of the people doing the searching, not the people being searched.
The people who want to be searched would have an incentive to make a quality site that the search service would believe would please their customers.
I can think of people willing to pay for quality searches - professionals looking for things they need, like programmers, lawyers, researchers, etc.
>The interesting thing would be coming up with a sustainable business model for it.
Even though I loathe ads, I wouldn't mind one simple, clearly designated ad spawned from keywords in the search. No tracking and no cross site linking. No result promoting, etc...
And yes, I believe that the current iteration of the web requires human moderation to be usefully searched.
Humans can also game systems to promote their garbage if they care to. Spammers can hire a click farm to privilege garbage results. Spammers and scammers seem to see enough returns to invest in ways to game the internet’s openness. There would need to be some kind of trust system to make the curation trustworthy.
Firefox on Linux Mint was pointing at something else for a while (DDG I think? Bing? I don't recall).
I gave up after a few weeks and had to switch it back to Google. Google's not perfect - it's never been perfect, it was just better than the alternatives - but it's still less bad than others.
Just tried andisearch and am extremely impressed. It has so far handled all queries I have thrown at it better than brave search and DDG. Will continue to experiment, best of luck and awesome work!
Thank you! I really do sincerely appreciate that. If you'd like to, please reach out to us there because we'd love to chat more. It wasn't my intention to take the discussion off-topic here because this thread is so incredibly interesting, but at the same time it's super encouraging to see the unprompted feedback and questions, and we'll share what we're doing properly on HN soon so we can address questions properly.
Distribution will come if the product is better, but it is a hard problem. I try every new search engine I can and they are always worse/slower than google.
I have tried using DuckDuckGo as my default search engine, but Firefox changed it back to Google with every update, so eventually I just gave up on that endeavor.
That's really strange, I've got it set as the default in Firefox on 3 different computers (2 mac 1 windows) and it's stayed the default over several updates. I think something might be wrong with your computer?
With the current Mozilla leadership it never even occurred to me that it could be a bug. I just assumed that it was something they do on purpose to get more money from Google.
>Let's be blunt here - almost no consumer consciously chooses to use Google search anymore
Do you have anything substantive to support this? I highly doubt it is true given the fact that the verb "to google" literally means "to search the internet".
I think you missed the point here -- people synonymize googling with searching and therefore aren't choosing to google -- they're choosing to search but ending up using google despite having made no conscious effort to do so (it's just there).
Google is the default search on the vast majority of phones and desktop browsers by default.
People don't change their search engine from something else to Google, because it is already the default search engine on the devices they buy and the web browsers they use.
So people do not make a conscious choice to use Google. The vast majority make no choice at all. Google is synonymous with search because it is already the search engine on their phones and computers. They are simply never asked which search engine they want to use.
Most consumers have no idea that you can even change your search engine. After talking with hundreds of users, they find it's either impossible to change (iPhone/MacOS) or too hard (Chrome).
If you're Duck Duck Go or Bing, at least you're in a very limited dropdown list if someone does want to try something else. If you're a new search engine startup, you're not an option at all.
Your argument supports the original poster. It is no longer a conscious choice, "Do I search for this via Google? Maybe I should use Bing? What about DuckDuckGo?", it is, "Oh, lemme Google that".
the other day, on HN i mentioned i was trying to find some relevant meme on DDG, and someone said "try googling 'foo bar baz'" and i thought it was funny.
I don't use google search if i can avoid it. I'll try 3 others first, and generally just give up. Google doesn't deserve any money.
> It is in all practical senses impossible for an iPhone or Mac user to change their search engine to a new search engine on Safari or at the iOS level.
There are five (very simply accessible) different choices for Safari on iOS.
But if you switch to iCabMobile on iOS there are TWENTY-FIVE search engines to choose from.
I think it's reasonable to point out that is not something most consumers are going to be able to do. The only meaningful search engine choice is that available within Safari. And you did install another App, they still aren't used from the system search on iOS, or from Safari itself.
I think you might as well be asking regular consumers to root their device so they can use whatever Apps from outside the App Store, or whatever search engine they want.
Also, even for a technical user, there is simply no way on an iPhone to change to a new search engine not already on a tiny list, and from talking with hundreds of consumers, I have not talked with a single non-technical person who could work out themselves how to change their Safari search engine to even one of the 5 limited choices, let alone a new option.
Unfortunately, installing an App doesn't let you change the system-wide search on iOS (or Safari browser), so rooting the device would be the only real way. My intended point is that if you're a consumer trying to change your search engine to a new option on an iPhone, there is no way to do it.
> This is the most annoying behavior because I really mean what I write.
Tons of people don't, though. They type whatever unprocessed half-second thought they have into Google and expect Google to lead them to the water, even if they're tugging and trying to go in the completely wrong direction. Google has optimized for working 'most of the time' for 'the most people', and that means striving for fixing the complete word soup of search results people type in.
A single mediocre experience optimized to work ‘most of the time’ for ‘most people’ is quite contrary to the narrative that has made Google such tremendous amounts of money (“let us surveil you so that you can have a more personalized experience”) though, isn’t it?
Given all of the data collected about Google users, ought not one of the applications of that data be some way to give users specifically what they are searching for if their past behavior suggests that they mean what they type? Couldn’t the “search only for <exact query>“ option be a very good data point on making that determination automatically, or enabling a user setting for “give me exact results based on what I actually typed by default”?
It seems possible to me that this behavior has more to do with the value of ads for “big” keywords than with (poorly) inferring user intent.
I have a sense that this is the dirty little secret of the spyware advertising industry, personalization just isn't that great. Yeah, putting you into a male or female bucket, parent or child, homeowner or renter, that's worth a little bit. But, to find out your name and address and search history and how long your last bowel movement took, just to deliver an ad that's theoretically hyper-optimized to make you buy something... I just don't believe it.
I don't believe that it's worth anything near what they are charging for it, except perhaps in the case of politics, which has always been an extremely efficient use of money. And even then, it's not worth a tiny fraction of the real cost it has to society.
Netflix has achieved the dream of movie studios going back more than a century now. They have the talent, the money, and more than two decades of data. Netflix knows what you watch, when you stop watching, how often you watch, which movie covers work best.
And yet, it's hard to look at Netflix as anything more than a total failure of the promises of data analytics and personalization. Netflix should be putting out nothing but hits. A dozen Breaking Bads or Game of Thrones.
Yet they are not. In fact, they do not even have a single show that is to the level of Mad Men, Breaking Bad, or The Wire. HBO and AMC are running laps around Netflix. Meanwhile, Netflix is making live action Cowboy Bebop and cancelling it before people even know it existed. I'm really curious what the data said about funding that particular project. On one hand, you have the cult following of the anime that will absolutely tear a live action version to shreds. On the other hand, you have to convince the uninitiated into viewing a remake of 23 year old anime.
Then there is the personalization. The fact that there is a meme about spending more time browsing the Netflix catalog than watching content tells you everything you need to know about how little people trust Netflix recommendations. Their new "top 10" feature is just depressing most of the time. It looks like a list of ten random DVDs in the bargain bin near the Walmart checkout line. Oh, and, their top 10 feature is currently the biggest recommendation feature on their site. And it's not even personalized! If that's not a complete admission of defeat I don't know what is.
Netflix has produced a lot of fairly solid content. Not at the level of the all-time great prestige TV shows like the ones you mentioned, but enough to keep a lot of subscribers happy for long periods of time at an accessible price. House of Cards (at least until the Kevin Spacey scandal blew up), Stranger Things, Orange is the New Black, BoJack Horseman, Disenchantment, etc are a few that come to mind that I watched and enjoyed.
I think Netflix has two big issues. The first is the way they drop seasons all at once prevents the natural cycle of pre-episode hype, post-episode interviews, speculation and leaks, fan anticipation, fan arguments (ship wars, etc), etc. Fan culture can't develop around this content as easily because there's never any breathing space for fans to collectively sit with the story so far. The shows that become a cultural force like Breaking Bad or Game of Thrones need us to keep coming back to the conversation every week. They have us talking about what happened last week and will happen this week with our colleagues at work and around the dinner table. How can these Netflix IPs enter stable orbit in the cultural zeitgeist when they're once-a-year events? It devalues the work, positioning it more like a movie that you watch and then forget rather than a story you become invested in over a long period of time.
Full season drops also allow people to binge a whole season of content for a single month's subscription and then immediately churn (ask me how I know). I assume they have some data-driven reasons for doing this, but it makes absolutely no sense to me.
Besides that, I think they should put more wood behind fewer arrows. They've developed a reputation for aggressively cancelling smaller shows with passionate followings which makes a lot of people not even want to bother until something has become an established mainstream success. I think of them now as the Google of content-creation, putting out a lot of solid (but not amazing) products and then cancelling them once people start to grow attached.
That's not how the entertainment business works. If I had to guess, Netflix's data-driven approach to content production is like card-counting in blackjack. It only gives them a slight edge on the house. A net positive outcome over hundreds or thousands of hands but offering no guarantee over the outcome of any single hand.
> HBO and AMC are running laps around Netflix
Netflix did win the most Emmys in 2021.[1] And they only started producing original content in 2013 or so. That's pretty good.
Wrote something similar before:
"The writing has been on the wall for some time:
1. Grading system changed from 1-5 to 1-2 (thumbs up/down). They thought that the users where full of crap when rating. I do believe some bosses just looked "bad" when buying in the next Adam Sandler movie. This started a cozy culture where no one in Netflix was wrong. Recommendation engine becomes comically bad, even with the best and the brightest.
2. They started to buy everything under the sun. South park made an episode about it even. All the comedians got their own stand up specials. It was now way easier to get a top score (thumbs up). Bosses where happy.
3. As they no longer focuses on quality which they no longer can measure (measuring time watched and churn is not that useful!), they start to strive for quantity. Which is expensive, very expensive.
I guess that in the next decade Netflix will become the next Comcast and cost 35 USD per month, and it all started in an innocent change to the grading system."
> Then there is the personalization. The fact that there is a meme about spending more time browsing the Netflix catalog than watching content tells you everything you need to know about how little people trust Netflix recommendations. Their new "top 10" feature is just depressing most of the time. It looks like a list of ten random DVDs in the bargain bin near the Walmart checkout line.
They may also be optimizing for revenues as opposed to recommendation quality (homegrown content being cheaper than licensed)
Part of this might just be account switching issues though. If I’m watching for myself I can usually find what I want quickly. The problem comes when I’m trying to browse with my wife to find something we can both live with. At best it gives the my preferences and the Union of me + wife’s preferences (and vice versa on her account.) But what we’d really want is a separate recommendation feed that shows the intercept of me + wife’s preferences.
But most people are lazy and won’t account switch for different contexts like that anyway, so there’s just no way they can keep the profile data as clean as it needs to be for a television.
Netflix is not recommending what they think you'll like, they are recommending what they want you to watch. Once you're a subscriber, they want to keep you there as cheaply as possible.
This is exactly what their data analytics has told them to do.
Have a few popular, quality tv shows with star-studded casts as loss-leaders to bring in new viewers. Otherwise the model is to produce and recommend the shows that get them the most eyeballs per dollar; the bare-minimum to keep their subscribers there:
- Stand up specials are dirt-cheap, quite popular, and provide never-ending variety.
- Ditto with 'reality' shows, bake-offs, make-offs, expose documentaries, etc.
- Old sitcoms and b-movies that have a proven re-watch-rate.
Throw in a handful of first seasons to keep the FOMO up, and you've got a captive audience on the cheap. Maybe one or two will catch on and become the next loss-leaders.
They may not have the quality shows that are 'running laps around' HBO and AMC, but by any of the metrics Netflix cares about they are simply running laps around HBO, AMC and everyone else.
>And yet, it's hard to look at Netflix as anything more than a total failure of the promises of data analytics and personalization. Netflix should be putting out nothing but hits. A dozen Breaking Bads or Game of Thrones.
I don't follow this argument. Knowing what people like has very little to do with the quality of original creative content; surely you don't expect Goodreads to put out Shakespearean novels, or Spotify to be producing original hits on par with the Rolling Stones? Should ESPN have better pro sports scouting and coaching talent than the professional leagues?
Knowing what people like, however, _does_ have to do quite a bit with selling those people a product - which Netflix just reported 15% YoY growth to $7.7B yearly revenue, they're clearly very successful to this end. I think you actually have it backwards - if anything, Netflix represents a total fulfillment of the promises of data analytics and personalization. Despite mediocre original content, this is a $200B company with 200M subscribers growing revenues by double digits two decades after IPO.
If Netflix paid $450k+ salaries to screenwriters instead of engineers, you'd very likely get better movies on a worse streaming platform. And when Netflix has shelled out for Hollywood talent, like Mindhunter which has David Fincher and Charlize Theron, the results are quite good.
Regardless, to take the fact that Netflix pays for premium engineering and analytics talent, but does not pay for premium filmmaking talent, and then spin that fact into Netflix being "a total failure of the promises of data analytics and personalization" is a questionable criticism.
Strong comment. I agree completely with your negative assessment of the “value” of consumer habits to optimize Netflix recommendation. In my own case I feel trapped in a very shallow local minimum. Yes I watched a revenge flick or two but now I am type-cast for life.
Netflix really doesn't need to produce something like the wire or mad men or breaking bad. There's no reason to make a show that appeals strongly to 70% of the market when they could make 70 shows that 1% of the market is fanatical about. They don't have only one channel that competes for content and they don't seem budget limited.
Ironically, it seems like AmazonPrime is far better at that.
As for then top ten being bottom barrel stuff, I think you overestimate how popular mad men was vs. something like king of queens.
I will say, Netflix seems to fail in many cases, and I don't understand how they think content discovery is supposed to work.
Right, people radically overestimate how much a profile is worth. Someone who owns a house in a rich area is somewhat easy to identify, and you target them... along with everyone else who is also trying to reach that rich slice, so you pay more.
The very high quality pieces of information can be things like "wants to buy a life insurance policy this week" or "just had a baby" or "just bought a plane ticket to XYZ," or "is in the frequent flyer program and spends more than $20,000 per year on travel."
However the majority of information about people, the overwhelming majority of whom have no significant disposable income, is worthless and not worth tracking for the most part. You reach those people through traditional mass marketing means.
> However the majority of information about people, the overwhelming majority of whom have no significant disposable income, is worthless...
I've had a supposition for a while now that the targeted advertising industry should be closing the consumer cashflow loop by advertising effective self- and employment improvement, with the objective of increasing the disposable income people have so they can then make more brokering traditional sales
It's a great point you have made. I work technically as a data scientist, but my domain is scientific data. I have quite a few GitHub packages and get recruiter calls for data science jobs almost every week, with pretty generous salary offers.
And from what it seems to me, there is a giant bubble. The vast majority of companies doing "data science" jobs are things that a smart undergrad can do with a month or two of training. And this is because I believe C-suites have completely gulped down the data is oil mantra. There are entirely charlatan companies with unicorn, even decacorn valuations now being built on this mantra - for example, CRED in India.
Yet, as you said, and as I believe too, most of the data is worthless.
Right, but unlike oil, most data is worthless most of the time.
For example, I'm about to list my house on the market. The real estate agents who reached out to me last year and the year before that to try to induce me to sell the house had no chance of succeeding. Now is the time if they knew the secret that I'm about the list the house that they should all be competing for my business. The only relevant piece of information is that I'm about to sell the house. That is a valuable lead that many in my region would bid on. My background information is frankly not that relevant to the value of that lead, and isn't something that is readily surfaced by the kind of deep profiling that is supposed to be going on. It can be signaled by me signing up to some kind of list that sells my lead to a zillion people at once, but is never going to be surfaced accurately to the people who can earn the most profits from it by my Youtube habits or whatever.
Zillow sells the logged in user data in the market I'm buying in to the real estate agents listing the houses, but that again is not something being modeled by some kind of big data operation, but is merely the same kind of "little data" provided on things like dating websites or LinkedIn when people browse your profile. There's no modeling going on there that requires sophistication.
>I have a sense that this is the dirty little secret of the spyware advertising industry, personalization just isn't that great.
Personalized adverts and recommendations can be incredibly, horrendously dumb.
Here's what I see when I hit amazon's homepage at the moment :
A "buy once again" column that features blackout curtains I bought 3 months ago (no, curtains don't need to be replaced every months, amazon.), USB cables I bought multiples of in the same time frame, a wireless charger (I already bought two before).
An entire line dedicated to showing me backpacks (I bought one less than a year ago)
An entire line dedicated to headphones (I recently bought wireless IEMs)
An entire line dedicated to watches (same)
I don't get it. Supposedly the best and brightest work at firms like amazon and google to brainwash us to buy stuff, but classic, random, non-targeted advertisement is more likely to make me discover products I'd buy than targeted advertisement because the latter only shows me things after I don't need to buy them anymore!
Here's what I would expect actually intelligent targeted advertising to do :
After buying a smartphone, recommend accessories (cases, screen protectors, USB-C dongles, chargers, whatever)
Here's what targeted advertisement actually does :
show me smartphones ads everywhere I go after I already selected and BOUGHT a smartphone. No, I don't need to buy another smartphone weeks after a recent replacement, amazon!
The same sort of phenomenon can happen after google locks on searches I did to buy something. I can't wait to see the internet advertisement industry crash and burn, it's overvalued nonsense.
A "buy once again" column that features blackout curtains I bought 3 months ago no, curtains don't need to be replaced every months, amazon.),
Disagree there. About 75% of the things I buy on Amazon are repeating purchases that I nevertheless don't want to be automatically scheduled. It used to be a real pain in the neck to reorder something manually, so I'm glad they made that easier.
But yes, in general, Amazon is full of low-hanging fruit that's been neglected on the tree for a decade or more. Buying clothes from Amazon still manages to be a worse experience than going to the mall, for instance, which is really saying something.
I think the fact that most recommendation algorithms have seemingly converged on what seems like a really poor and naive implementation - fixation on very recent activity - shows that the sort of deep personalization touted is mostly BS.
Both YouTube and Amazon heavily personalize by recommending primarily the 3-4 things that I've interacted with in the very recent past.
This is not true. For example every time Summoning Salt uploads a video, which happens every few months, it will show up on my recommend feed because YouTube knows I'm willing to watch their ~1 hour documentaries even though I'm not subscribed to them.
Youtube seems to be a rare exception here in that people actually feel like its algorithm is useful. However, even then, their algorithm mostly seems to devolve to "what creators have you usually watched videos from" and (usually directly after you watch such a video) "what videos did other people who watched that video watch?" Basically the same principle as PageRank, just with a lot less spam to deal with.
This could (probably isn't) be a very quick implementation with a heuristic like 'if percentage of viewed videos from channel x (essentially per channel viewed) > threshold ==> show new video from channel x on homepage next time user appears.
Make it fancy and use a multi armed bandit and call it machine learning/AI/data science.
I haven't consumed significant amounts of ads in a long time, only some logos in sports and the occasional visit to family or the rare times adblock fails (YouTube premium user too). So I can only imagine how hilarious that must be.
I think you're right. I'd like to see an analysis of the effectiveness of personalised advertising based on tracking versus ads based purely on local context. The latter being if you're on a web page about birds then you get ads for bird seed and bird houses. No tracking involved.
Does it really work so well in politics? I've read in various places that a lot of political advertising in America functions basically as a means for channeling donors' money to a few K Street firms belonging to party insiders.
It's pretty straightforward to understand that when the vast majority of your sites income is generated from ad revenue, that data is being used to optimize for generating ad clicks, etc., rather than actually giving users the best/most relevant/useful/desireable information for their purposes.
I agree with you. I highly doubt that our economy has enough (product, message) combinations to justify the need for personalization based on more than a dozen attributes.
I will buy X, if I need X. And once I buy X, it's done. For example, I wanted a cordless drill last week. Did the "site:reddit.com" thing (I actually have been doing that almost subconsciously now, as Google results are all trash), chose a drill, and ordered one off Amazon.
Then, after that, what's the point in showing drill ads to me for two weeks?
There's a well known effect in advertising that advertising a product to a person that has already bought that product generally increases their satisfaction with the product and the purchase, and may cause them to recommend the product to others.
Probably that's what they are going for if they're doing it on purpose.
> There's a well known effect in advertising that advertising a product to a person that has already bought that product generally increases their satisfaction with the product and the purchase, and may cause them to recommend the product to others.
Do you have a link for further reading on that? That's fascinating if true.
It's not supposed to feel good. If 9/10 people have a brief negative thought about the advertising experience and nothing else happens, but 1/10 people happen to have their friend on the phone at the time and makes a referral, then overall that is a win for the brand.
From the behavior of ads (that I imagine are highly optimized), all that knowledge is useful for front-running an specific TV model all over your internet once you decide to buy a TV.
It seems to be completely useless for anything else, and specifically harmful for product discovery, that is the one way ads add societal value.
The most common pattern I see relating to personalized advertising as someone being advertised to is that I will often see an ad for something I just bought (or some competitor to it) repeated relentlessly for a couple of days after buying it and this is after not seeing any related ads during the days prior where I was actually doing some research into the product space.
Maybe I'm an outlier but they seem to miss the window of relevance on me often enough that I notice it as a commonly repeated pattern.
I know it seems moronic, but I think it might actually make sense from the advertisers point of view. Some percentage of people who buy a thing are going to return it and buy something similar in the next week. That percentage is almost certainly large compared to the percentage of the overall population who's going to buy that thing in the next week, and it seems plausible to me it's even large compared to the number of people who have been browsing for the thing but haven't bought yet. (Think of it as the ratio of people just browsing vs ready to buy.)
Even so, wouldn't it be much smarter if they kept track of what the expected life expectancy of the thing you bought is, and then years later start feeding you ads for a replacement? Or is it too hard to track people over such a long period of time?
I don't think any advertisers would be able to offer "People who bought a washing machine 3 years ago" as a category that can be targeted without a riot
Same experience. What's even more mystifying is that often it is for items that no human would be likely to be buying many copies of in a short span of time (high ticket items, or items where you probably don't need more than one).
Just because I bought something doesn't mean I kept it. And those 0.1%, or whatever, returning items are very likely to buy another one of a different brand.
Exactly. If I just bought some power tool for a home improvement project, I am the least likely person in the country to want to buy that exact same power tool the next day.
Not if you hate it and want to return it. In fact there’s a calculation to be made - what percentage of people return or dislike their drill? Because that subset of the population is probably more likely to be looking to buy one than any other.
A return rate of say 1% may lead to more people looking to buy a drill who have just bought one in the last week than people looking to buy their first drill.
> In fact there’s a calculation to be made - what percentage of people return or dislike their drill?
If that's true, they left something out of their calculation: What percentage of people will install an adblocker as a result of feeling like they're being hounded for a few weeks? This scenario was mentioned specifically by Tim Cook when he introduced the Safari anti-tracking features.
It's always fun watching an ad system try to figure out nonbinary people. Spotify ads can't decide whether I'm a successful businessman or Spanish-speaking housewife.
It's always fun checking Google's ad settings and seeing what they think I'm into.
Apparently now I'm into baseball, flowers, boating, celebrities, country music, credit cards, geology, event ticket sales, fishing, and windows OS. Among a couple hundred other things. It even gets some rather basic facts (marital status, company size, education) wrong. I seriously wonder how they generate this profile?
Well... It actually got some categories right, but I don't really feel that's very impressive considering that it put me in every category by the looks of it.
It fascinates me to see how the ad algorithm responds to people who watch content in multiple languages. I study a lot of languages as a hobby, so I often watch YouTube videos that are in Mandarin, like news broadcasts and niche hobby channels. YouTube has now started showing me ads (in Mandarin) which seem to be targeted to Mandarin-speaking immigrant parents of young children who want a way to teach them Mandarin despite my, and my spouse’s, very busy careers. I find this amusing because I am a single, pasty white man in my 20s.
Some people fit in convenient buckets, but lots of people don't, and assuming all people do, will make the ad system useless to a lot of people. Even if you're not non-binary at all, you could still be a successful businesswoman or a Spanish-speaking houseman (househusband? stay-at-home dad?).
Better to just follow people's interests, instead of using their interests to incorrectly pigeonhole them and then drawing incorrect generalisations from that.
That’s just Spotify. Many years ago they had a little tool that actually reported what demographic slots it pegged you at based on your listening preferences. The top two hits for me were 1.) early 20s, college educated, White, woman 2.) 60+, blue collar, African American, male
At the time I was a late 20s, college educated, South Asian male. I’m very cis and very straight. And yeah my musical tastes are pretty eclectic, but that was a weird profile to settle me on.
Something about my actual interests and activity apparently makes youtube think I'm into Fox news and all the crazy shit found there.
Now, who else has this same value judgement about me? This assessment that I neither declared for myself nor even ratified.
It's annoying but ultimately harmless that youtube shows me conservative wackjob stuff.
But is that same profile in someone else's database that marks me as someone to watch or something? Does it affect my insurance rates, my liklihood to get extra scrutiny when travelling, my ability to purchase or register a firearm, my access to jobs that might be extra sensitive or responsible, basically any of the things where someone either private or the state does any sort of background or credit check on you for any reason, and there are really many of those when you think a out it.
I'm guessing, today, it's probably not really affecting my life in any real way, but, there is no way it makes any sense to say that will still be true tomorrow.
There was that infamous case of a retailer figuring out someone was pregnant before they did based on what they were buying and mailing a customized flyer...to their dad's house. I don't remember the exact situation, but it probably wasn't the only incident.
I believe you're referring to a rumor (which may be true, I just mean it in the sense that it's out there and not something you or I have verified) about Target.
That btw is an anecdote from the association mining community.
I spent a lot of time learning about association rule mining in my AI courses, including the implementation details of competing ways to mine them. The technique seems extremely useful and fascinating (I jury rigged it for on the fly league of legends champ recommendations to maximize calculated win rate change given limited information), but I almost never see it used in the real world or even see it talked about anymore.
My experience working in a similar domain (NLP summarization, which leverages methods like text rank which are identical to pagerank but for text summation) is similar.
Personalized page rank is not significantly better at summarization in my experience, even "queryable" summarization, but that also could be a pure implementation problem or a problem of hyperparamater selection...
I worked for a healthcare recruitment company in a capital city with some large hospitals and a number of universities. I can't for the life of me understand why they chose to spend so much money on trying to track healthcare professionals online when they could just advertise it on-premise where they actually hang out.
I can't understand why every company want me offer “personalized” service. It never works, and if I can't manually set preferences, then it's not personalized (because “personalization” means exactly that).
Given all of the data collected about Google users, ought not one of the applications of that data be some way to give users specifically what they are searching for...
You're missing what "personalization" has come to really mean. It means knowing enough about the user to give them an experience you can profit from and which they will accept. If there isn't something you can expect profit from, there's no reason to give them anything.
This used to be solved by allowing queries like `Class Inheritance +ruby' to require results to include "ruby". They killed this for Google+ by changing it to quotes, so `Class Inheritance "ruby"' but now they interpret even those. When I use Google, which is less and less, I am not looking for a fight with a computer to express my intent, I'm looking for the answer to a question. That never seemed to be an issue until recently.
I work for Google Search. If you put a word or a phrase in quotes, we will only find things that have that exact word or phrase. Nothing has changed in this. When it happens that people feel it fails, it's often that they don't realize we've matched that word or phrase appearing in ALT text or text that's appearing in a less visible part of the page -- or in a few cases, the page might have changed since we indexed it.
> If you put a word or a phrase in quotes, we will only find things that have that exact word or phrase.
I'm sorry to tell you, but this is flat wrong. I commented[1] about this a few months back with a random phrase as an example. I see it often in my day to day also.
I searched for "eggzackly this" and all 10 results on the first page contain the phrase, although most have punctuation in the middle.
Looking at all 22 results (without opening the "omitted results" section or image results), the phrase became harder to find off the first page, but I tended to find it in the source code, or the DOM, or the cached version of the page, or by disabling JS (sometimes requiring a combination of those techniques). I found it in 21 out of the 22. The one page I couldn't find it in (the dolls one) looks like a frequently updating page, and it was highlighted in the snippet, so it looks like it was there at the time of crawling but not now; the cache isn't there for that page.
Full disclosure I work at Google, but not on Search.
The article has been updated with a response from Danny Sullivan, the person you're responding to, which is worth a read.
I just tested "eggzackly this" and every result on the first page contains the string "eggzackly this", albeit usually with different punctuation ("Eggzackly. This [...]", "Eggzackly, this.", "Eggzackly! This [...]")
Punctuation matters. I explained this in another response, and I should have mentioned it as part of my response here. But to repeat, these are typical reasons why it might seem that quoted search isn't matching when it is:
1) text appears in ALT text
2) text is not readily visible on a page (maybe in a menu bar or small text)
3) there's punctuation ("dog cat" will match "dog, cat"
4) page has changed after we've indexed it (so view the cached copy, if available)
In the [eggzackly this], you found matches of those words separated by punctuation -- which we interpret as a space, so the phrase is matched.
I wish we'd fix the situation with punctuation. I get that's confusing. But that's not a new change; quotes have operated that way for ages.
I'm sorry but this has absolutely changed. I'm not sure why but quite often we are suggested results in queries that ignore quotes. The engine is even telling us that if omitted those terms.
We don't have control over this and it's very frustrating.
We haven't changed anything. Promise. Honest. Not at all. But we definitely want to look into any cases where people feel this isn't working, so actual examples (if people are comfortable sharing) will really help.
What you're talking about is probably a case where there's a quoted word or phrase as well as other words that aren't quoted. In such a case, we're going to absolutely look for content that matches the quoted parts. That's a must. The other words, we'll look for them, but we'll also look for related words and sometimes, we might find content that doesn't match one of them.
Because those other words aren't quoted, we'll tell you if we find a match that seems helpful but doesn't contain those non-quoted words. That's what the message is about. But it should never be telling you we omitted a quoted word or phrase because we won't -- with one exception.
If there's literally nothing on the web we know of that matches a quoted word or phrase, then we're not going to show anything at all and say we couldn't match any documents.
I tried to find a counterexample and I couldn't! I believe you that quotes really are working. What's confusing though is that a quoted word or phrase often doesn't show up in the Google results snippets. This is certainly the reason why people think quotes aren't being respected.
Though, why does enabling "Verbatim" (in tools) on a search reduce the number of search results if all my terms are already quoted? Enabling Verbatim often does it make it feels like my queries are interpreted more literally, but if quotes are already being respected I don't understand how Verbatim would reduce the number of search results.
I agree, it would be easier if it were in the snippet. That's something we're looking at. I believe it used to, but sometimes the quoted part might not have been the best overall snippet to use. But as said, we might revisit that.
On the counts -- basically, it's all really rough estimates. We make a rough quick count, you go deeper into the page, we make a fresh estimate. It can change, and it doesn't always make sense and personally, I'd hope we just get rid of counts because of this, perhaps more confusing than helpful.
Interestingly I tried my pet peeve search, and it worked for the first time this year! It is for a specific recipe, and I search for 'ocau slow cooked balsamic beef'. I have had to manually find it in the archive of the overclockers.com.au forum for the past year, as Google seemingly forgot it existed no matter what search terms or operators I used.
The main difference is I am using Firefox on Manjaro and not my historically typical environment for searching. Normally I would either be using Chrome on Windows or Chrome on iOS if I am in the kitchen.
I will play around on some other devices and see what evidence I can find.
P.s. I'm being referred in for a role at Google currently, is the Search team only in a specific area like Silicon Valley or is it a global team? Most of the jobs in Australia seem to be commercial facing, not product facing.
Glad to hear that works! Search has teams around the world. I'd suggest if you see something relevant, apply even if it's in a particular location. Remote work has changed a lot things.
This is not the case in my experience. I type a query with some parts in quotes and often get lots of results that have in small letters at the bottom something along the lines of “does not include <word in quotes>”, with no in bold highlighted part showing the phrase in the page context. This was not the case in the past and google made sure the word I put in quotes is absolutely mentioned somewhere
I’m guessing this happens when there are less results matching my phrase
I would love if you or anyone who ever has this happen can share an example, if you're comfortable doing so. We'll debug. But if you quote something, we shouldn't show anything but that which matches the quoted material.
Now, if you quote something and put in other non-quoted words, then we'll look for stuff that matches the quoted part and the other things are optional. So when you see that strikeout message, it means basically "We found this page that has the exact words you quoted, and it probably has one or more of the other words or related words you didn't quote, but heads-up, it doesn't have one of those non-quoted words at all."
And we do this because sometimes there might be a useful page that doesn't contain all of your optional non-quoted words.
Totally agree it would help if we did a better job bolding the sections of a page where the quoted terms apply. Often we do, but sometimes the snippeting won't include them if there's better text to describe the page overall. But we're looking at maybe improving here.
Unless it's changed in the past 56 minutes, the SERP is pretty up-front that it can't find any pages with all three of those.
It literally says `No results found for "kitchen boss" "g320" "disassembly".` right at the top, and then shows you the (properly explained) results for the non-quoted version of the query.
Mobile doesn't show that, at all. It also doesn't show that bit if you use the brand name "kitchenboss" instead of "kitchen boss", which I originally intended. My phone might have auto-corrected that without me noticing.
Anyway, with the terms I posted earlier, on desktop it says:
Results for kitchen boss g320 disassembly (without quotes):
It looks like there aren't many great matches for your search
Then it goes on to show me lots of unrelated results, instead of showing me the 'not many' great matches for my search. If my search doesn't turn up many good results (let alone great), that fine. Show me what you've got, let me decide how to broaden my search to get more. All of the unrelated results being shown are marketing spam sites. That's not helpful.
On mobile and desktop, I'm getting a message saying that when you search for ["kitchen boss" "g320" "disassembly"] that we have no results. And there's not much we can do if there are no pages we find that match all those words you required be present. There just aren't the pages.
What we can do is try searching for all of the words, so perhaps you'll find something useful that way. But even we can tell when doing this that the results might not be what you want, which is why that automatic warning about maybe these aren't great matches.
IE: we can't show you what we've got for a query where there are no exact matches. It's impossible. We can show you what we got if we don't require all the word be present. And we can tell you what we're doing. And you always have the choice to restart the query in another way if you don't like that.
Now here's something else. You probably used the quotes because I'm guessing you figured it was better to tell us exactly what to do than trust us to look at all the words and analyze the context and so on and see if we could make matches generally. If you had done that, just searched for [kitchen boss g320 disassembly], then the first web page result is the instruction manual for your sous vide machine. It has cleaning steps, which I'm guessing also might be what you're after? (It looks like it's just take off the outer casing).
Those results, doing it directly like that, are different than when we gave them to you after your quoted search failed. and that's probably because when you gave us quotes, and there were no matches, we might have tried some stricter matching to keep closer to the original requirements rather than use our general ranking.
To wrap up: maybe don't try the quotes at first. It's totally fine to type in a long natural language query like [how do I take apart a kitchen boss g320] and if you do that, the instruction manual is right there.
I added the quotes because I wasn't getting what I wanted without them. I don't need the user manual; I already have that and I'm not just trying to clean it. I wanted to find a guide to fully disassembling it, like the videos you find for phones when you want to replace a cracked screen yourself. Knowing that there are no matches at all is good information. (Disappointing, but good.) Trying to give me results for a different query, apparently assuming that I accidentally used the exact-match-only quotes, is not useful and kind of condescending. I think that's what leads people to think the quotes don't work the way they used to. Finally, the wording about not many good matches makes it sound like there are some good matches, but they've been mixed in with these other irrelevant matches. That's probably just a wording issue on the message, trying to soften the "We can't find anything" result.
Weird. On a computer, `"kitchenboss" "g320" "disassembly"` returns exactly six results, all of which appear to include the quoted terms. Plus the "It looks like there aren't many great matches for your search" message at the top. Which sounds like exactly what you want. I wonder why it's different on mobile.
I routinely see queries with quoted keywords where results don't have them highlighted in the snippets on the results page (but do have other, non-quoted keywords highlighted!).
That doesn't mean the quoted words weren't present in the content. It just means our system didn't think creating a snippet around those words was the most relevant snippet. Which I get, in some cases, actually would be better. It's something we're looking at.
If a snippet doesn't contain the words I searched for, I don't click it, because I assume Google has fucked up the search again and given me some irrelevant thing. If that is what has actually changed in the last few years making people think results are bad, the snippet algorithm, please revert that. I want to see the exact context that the words I typed appear in the page.
Your system for snippet creation is so bad that according to you it's created a situation where many, many users believe that quotes don't work because of how bad it is.
Please fix it, like now. Displace your teams current sprint priorities, or the anti-google search backlash will turn into a situation where in 2025 Google is on the defensive for search market share from a Phoenix rising yahoo or something like that.
The fact that anyone at Google ever okayed this behavior at all in the first place is simply rage inducing and you should see that with the magnitude and persistence of the "actually it really doesn't work bro" kind of comments.
So, besides the possibility that you are flat out wrong (as another commentator claims you may be), let's assume that you're right.
1. This is still horrific UX/UI
2. The culture internally at Google seems to have a "we know better than the users" attitude in all things.
3. Query rewriting is a horrific technique in general with almost zero value to life/society outside of fixing spelling errors. Whatever your A/B testing says about it's purported utility is polluted by Google's own dark patterns and political whims of the managers who run the internal search organizations.
It helps to actually know better than your users if you want to take the attitude in number 2. I don't believe that you or Google knows better than it's users, for many reasons previously enumerated in this thread and others.
One day Google search is going to be displaced and it's current utilization of query rewriting techniques will be one of the fundamental reasons for this.
You should take the absolutely massive amount of recent criticism and the fact that users repeatedly claim it's happening in the face of your claim it's not seriously rather than literally blaming the users writ large for a problem that is fundementally with the behavior of Google search.
I don't have any recent information on how google search works, but years ago it looked at the expertise level of the searcher. So newbies received newbie results, advanced searchers received advanced results (and more visibility into filtering functionality). Today... they're hiding the advanced features and also seem to be reducing personalization of results to save compute resources. It's horrible.
You: Class Inheritance +ruby
Google: searching for "cash inheritance..."
I work for Google Search -- we never operated like this. We don't know that someone is somehow a "newbie" vs and "advanced" searcher and change (nor did change) the results somehow.
This is very helpful if I search for a name I didn't quite pick up or don't know how to spell, or if I only remember fragments of a quote or topic, then I just blurt out my stream of consciousness and Google will mostly point me in the right direction. That being said, I wish I could explicitly tell Google to treat my query more literally. Ideally you would be able specify the search query in some kind of grammar. They have these kinds of prompt mechanics for GPT3, so I doesn't seem too unrealistic, even if it's all ML nowadays.
I work for Google Search. We have several ways for you to do this. The easies is to put quotes around a word or a phrase that you absolutely, positively want to be present in content retrieved. And yes -- it still works. It really really does, but if you or anyone finds an example where you believe it doesn't, please let me know. We'll debug it. The reasons people sometimes think it's not working is because the text appears in ALT text, or it appears in text that's not readily visible on a page (maybe in a menu bar or small text), or there's punctuation ("dog cat" will match "dog, cat") or sometimes a page has changed after we've indexed it (so view the cached copy, if available). You can also use verbatim mode from the toolbar so that we search for only the exact words you provide.
I don’t think op meant it literally, but the fact that the results are so keyword stuffed that despite “appearing” on the page they are actually irrelevant to the page and thus useless.
Have you tried this? It immediately turns verbatim off. For everyone! It's been like this forever. Years and years. Gaslighting won't help here!
In terms of quotes, again, this does NOT work like +, like verbatim. If it did, then that term would absolutely show up, just as it is, in search results. Yet, over the years I've seen:
- aliasing happen from within quotes (EG, bob -> robert)
- quotes entirely ignored (eg, those terms NOT showing up)
Yet searching with verbatim on, immediately causes those quoted words to appear!
You are absolutely gaslighting people on this! Right now, in this thread. And if it isn't intentional, if you aren't gaslighting, then how can you not even notice that verbatim turns off, the second you select a date range?
I want to say so much more here, but it's filled with such ... vitriol, that I think my terminal would melt.
I'm not trying to gaslight anyone. And what would be the point? To say something works if it demonstrably doesn't work?
Yes, I tried this for some of the presets before I replied. It worked. It still works.
My sincere apologies for not specifically testing custom date range option as well. I should have; you are correct. That won't work. I'll pass it on to see if there's a way it can. My apologies again.
If you need to do this another way, what I also said works. Do it in the search bar using the before/after command. Just quote all the words, and that's the same as verbatim.
It's like you're just spouting Google propaganda, without validating what you're saying. Just like with vertabim/date range, which has been a thing for close ot a decade, which Google has received endless reports about, this too does not do what you say.
Your responses are akin to those canned responses one gets when you post a bad review. "So sorry, please contact us here at this email address.", which of course is all about impression, and results in nothing ever happening.
This has been going on almost a decade, yet oh what, no, haha is your response.
Verbatim mode means we search for exactly the terms you put into the box. Quoting terms means we search for exactly the quoted terms. That's what I meant by them being them same.
So if you did this search in verbatim:
[search for this]
It's the same as:
["search" "for" "this"]
in terms of the instructions we're getting on what to retrieve -- look for content that has all of those words and only those words. No spell check. No synonyms. Just those words.
The ranking of the results might differ, because we probably use slightly different ranking systems when using verbatim versus quoted words. But in either case, the retrieval requirement is the same. Results should have all the required words.
I don't really see what the link above is saying to somehow refute all this. That link is about a quoted search for ["eggzackly this"] and nothing to do with verbatim mode. And it says that it found those two words in that order with punctuation...
Which is what I explained elsewhere in this (now huge) thread. But to give it again:
Typically the reasons people believe quotes are not working when they are is because:
1) text appears in ALT text
2) text is not readily visible on a page (maybe in a menu bar or small text)
3) there's punctuation ("dog cat" will match "dog, cat"
4) page has changed after we've indexed it (so view the cached copy, if available)
In the [eggzackly this] example, that's what was happening as the poster saw -- we found those words separated by punctuation, which we interpret as a space, so the phrase is matched.
Personally, I wish we'd fix the situation with punctuation. I get that's confusing. But that's not a new change; quotes have operated that way for ages.
Most important -- quotes SHOULD work as you and others are expecting. We WANT them to work that way. That's why we spend time looking at these reports saying they're not. I have spent lots of time doing just that. We find the matches. But if anyone believes they aren't working, and the reasons involved above aren't happening, let me know. We'll get on it. We want them to work as expected, and we want everyone to feel they're working that way.
For additional clarity, quoting has always, always, always been different than +, and verbatim. When you(Google) removed +, so that 'google+ searches' could work without interference, quoting was already a thing, and people were just told by some airhead googler "Oh, but quoting is the same! Just use that!"
It wasn't. It isn't. I never has been. Ever.
Verbatim was introduced to replicate that lost + functionality, after massive outrage at the inability to find search results. The fact that you, and other Googlers still think "" is the same as verbatim/+, when it doesn't even show the same search results, is highly, highly questionable.
To be beyond blunt, you're wrong. You are completely and totally wrong. +/verbatim and "" are not the same thing.
Please go away, and learn how your own product works, before commenting on it, ok?
At the time, + was used to require that something be present. Quotes were used to require words appear in a particular order. You could do a search where you quoted a phrase, but that didn't necessarily require it to be present (as I recall). If you absolutely wanted the quoted phrase to be there, you had to quote and put a + in front of the quoted phrase. So yes, they were different things.
When + was eliminated, quotes took its place. Quoting a single word was the same as when you used to + a word -- find the exact word. Quoting a phrase still meant find the phrase, but that also meant it was required to find the phrase.
With +, then with quotes and with verbatim, it's about what you retrieve. Verbatim says get these words or words and only those words. Quoting says get these words and only those words (and only those words in a particular order, if you indicate that). Just like + used to mean get these words and only these words.
The ranking of results might vary when you quote versus verbatim, but what you're asking to be retrieved is the same to us.
At the time, + was used to require that something be present
When + was eliminated, quotes took its place. Quoting a single word was the same as when you used to + a word -- find the exact word. Quoting a phrase still meant find the phrase, but that also meant it was required to find the phrase.
No, quoting did not take its place. At all. That's why verbatim was introduced, after outrage. Google claimed it did, but it still aliased. It still decided to provide results without quotes.
Again, this is why verbatim was born. From that "Google no longer gives precise results, ever" angst.
As for the quoting, it should work. And if you have an example where it's not, please let me know. Quoting is designed to exact match. It shouldn't produce synonym matches, correct misspellings, find content that's not on the page as we saw it when indexed (I think I did say one confusing part is that if there's punctuation, that gets dropped when matching).
I can confirm what bbarnett says about date ranges and verbatim. Turning on a date range toggles verbatim off. Turning on verbatim toggles the date range off. It's impossible to enable them both.
I read the article, and the HN comment it links to, but didn't find an example in either, and it doesn't match my experience. Does someone have a concrete example when using quotes results in pages not containing the search terms?
I work for Google Search, and as I shared elsewhere, quoting still works. It really does. If you or anyone finds an example where you believe it doesn't, please let me know, and we'll debug. Typically the reasons people believe it is not working is because:
1) text appears in ALT text
2) text is not readily visible on a page (maybe in a menu bar or small text)
3) there's punctuation ("dog cat" will match "dog, cat"
4) page has changed after we've indexed it (so view the cached copy, if available)
I believe you! (see my other comment in response to the original one).
But it seems people don't (my original comment is being heavily downvoted because of this). And although they can't submit even one example, the fact that they don't believe you is obviously a symptom of a bigger problem.
For some reason, Google is losing the trust of power users.
No results containing all your search terms were found.
- few results with quotes, not more without quotes:
Your search did not match any documents.
It looks like there aren't many great matches for your search
Tip: Try using words that might appear on the page you’re looking
for. For example, "cake recipes" instead of "how to make a cake."
- no results with quotes, but results without quotes: Google says that the search with quotes didn't find anything, and that they searched without quotes instead.
I have yet to find any instance where Google corrects the inside of quotes without any warning.
Google has been regularly ignoring quotes for at least 5 years, probably longer. That was one of the biggest factors for me dropping it as my main search engine.
That's like speaking to little children, that are learning to talk, reproducing their errors. Some adults believe that it's cute, but it's idiotic, confuses the babies and make their progress more difficult and slow.
I don’t think this means anything for the point you wanted to make about search results, but please note you’re exactly wrong about baby talk! It’s not a good analogy.
Baby talk (or CDS, child-directed speech) helps engage their attention and provides valuable feedback. Kids who experience less CDS develop language more slowly.
N = 3, we intentionally never baby talked to our kids, and spent a lot more time reading them novels and other things without pictures or simplified language than (I'd guess) most people do (we did also do plenty of picture books), and their language development was in all three cases way ahead of schedule.
Could just be luck (well, genetics, probably) I guess. Maybe they'd have developed even faster if we'd used baby talk. One shitty thing about parenting is it's really hard to tell what helped, what hurt, and what didn't matter at all.
>...their language development was in all three cases way ahead of schedule.
I would put a lot more of it to having (seemingly) engaged parents. Even a backwards strategy enacted by a loving parent who is consistently trying their best is likely to outperform the result that most can manage (owing to time/money/education/etc).
That is my belief as well: being there, listening, interacting lovingly, paying attention is overwhelmingly more important than a particular technique.
I've never baby-talked to our son, but I do coach him to say things that are within (or almost) within his speaking capabilities. So for instance, this evening we were reading The Gruffalo, and he pointed to the fox and said, "Fox eat!" I said, "The fox wants to eat the mouse?" He said, "Yeah!" So I tried to coach him to say "Fox eat mouse". He got as far as "Fox eat there"; maybe he'll get to "Fox eat mouse" in a week or two.
It's not genetics or a language strategy, just engagement.
Compare a toddler living in a ghetto concrete jungle to one who takes daily strolls through Central Park or Suburbia. The enrichment of parents teaching about the trucks and the trees and the lady with a purple hat pays massive returns.
I did this as well with the same results (but also have reason to believe genetics played a major part). But I'm not sure we're optimizing for the right thing. I'm far from convinced that accelerated language development is a good thing. I think development may suffer in other areas.
Why would you believe genetics plays a part? There is minimal evidence for that. You have actual evidence for things like your higher than average time engagement, nutritional indicators, as well as health and dental care. You probably live somewhere with decent air and water quality. Then of course the likely fact that parents have relatively prodigious vocabularies, fluency and articulation. This is why your kids are smart.
Lots of evidence that genetics have a large influence on early language development, especially speech, if you care to look. E.g. this study finds genetics contributing over 60% of variance: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3851292/
I did goo goo gaa gaa for the first 6-8 months since you can tell there’s really nobody home up there yet and it’s cute and engaged my other kids to play with the youngest. But yes. Mine seem fine as well so I doubt CDS is going to make/break a human being.
Fathers descend to "baby talk" when the child is learning and slowly bring them up to par instead of trying to just force perfect talk from the start. They do this instinctively.
There's some great comments on this from salman khan, I think. He recorded the first years of his kid's life at home and documented this phenomenon
You think the baby really wants to hear googoo gaagaa? Now, they are trying to say "I'm hungry. Feed me!" Babies must look at adults doing the googoo gaagaa, and think to themselves that these adults are absolute morons.
The sites that Googs returns are basically the internet's version of googoo gaagaa. I look at the websites returned, and often think that the site's owners must be morons. Useless drivel clearly designed to game the Goog search results. I think think about how moronic it is that Googs allows this.
You think the baby really wants to hear googoo gaagaa? Now, they are trying to say "I'm hungry. Feed me!"
I suspect they know the sound they want to make but they don't know how to articulate it. They make an approximation and we can encourage them repeating the correct version, so they realize we understood what they're trying to do: "you're half way" but repeating their approximation is misleading.
In my experience waaaaaaaAAAAaahhh meant wet diaper or something hurts. Leeeeeah, leeeaaaahhh was being hungry. Phonetically is similar to the beginning of a polish word mleko (milk)
I honestly find it pretty helpful. You can type "russian murder painting" into Google and it will come up with Ivan the Terrible and His Son. All that hinting may be annoying if you know exactly what you wanted, but I'm not a specialist in everything I ever search for.
Then again, both DuckDuckGo and Kagi also give that result for that search phrase. As well as being more generally useful for more specific searches as well.
What would be nice is if you could toggle this behaviour. Sometimes I know exactly what I'm looking for, sometimes I don't. Assuming I never do is at least as silly as assuming I always do. Just give me the option.
I am frankly baffled that after all this focus on "personalised search", they still don't actually allow you to personalise your search like that.
Right, although piping junk into the search box and expecting it to bring back something useful is trained behavior.
I've been using DuckDuckGo a lot more recently and the thing that surprises me isn't the kind and quality of the results, it's that I actually need to use my brain to search.
It's not about whether this is a good or a bad thing—I kind of like the precision in a way, it's just jarring how different it is as an experience.
It's funny to observe my stepson learning his way through Google. It's happening mostly through the assistants on TVs and locked cellphones. But he's learning to do exactly what you said: half-second thoughts and brute forcing many queries for the same subject. He's 7.
A less charitable interpretation --- and unfortunately one that could be true --- is that Google does not want you to think. It wants to keep you stupid because it's easier to deceive those who can't think and bend their thoughts in the direction that gives G more $$$. I'd say it's not merely optimising for the stupid; it's actively encouraging it. It wants to be your brain, control your thoughts and life.
I see what you are saying but it seems to me that it used to do a much better job at that. These days I feel like I'm fighting the search engine constantly and it is certainly not magically finding what I want anymore. It feels like some crusty unmaintained tool that I have to know how to use.
Google has optimized to whatever sequence of behaviors achieves the most profit. The search results are not chosen for utility to the user but as nudges in a cycle of influence intended to drive you to attend to an ad, purchase something, or consume particular content.
They should not be engaged in non-consensual manipulation of social or political behaviors, and the ethics of market manipulation at scale through advertisement are far from clear.
I use market manipulation to mean just that, someone manipulating the market in whatever way. I'm not familiar with the legally-oriented meaning of this term.
Market cornering is classic market manipulation. Google uses every asset at their disposal to maintain their 98%+ death grip on the search markets. The list of competitors bought, stifled, legally crushed, or absorbed is probably endless. The search market is thoroughly cornered.
I used the phrase intentionally and specifically. Advertising isn't always market manipulation, but it can be and is used to that purpose.
Google uses advertisement and content "curation" to manipulate consumers. This results in product preference, purchasing behavior, and market conditions favorable to Google and/or unfavorable to Google's competition. This includes siloing consumers in political bubbles and manipulation of narratives through the deliberate selection, order, and pacing of content exposure based on the intent of Google's shotcallers.
The reinforcement cycles inherent to their algorithms are used to manage the information made available to vast numbers of people, with highly detailed behavioral profiles used to achieve behavioral outcomes, whether it's buying something, voting, or preferences for or against particular policies or candidates.
Do they? I see this stated all the time, with no references.
They type whatever unprocessed half-second thought they have into Google and expect Google to lead them to the water
Perhaps if Google didn't try to fix things for people, they would be more thoughtful with their searches.
Take away the junk food, and people will resort to real food. The same way some cities limit parking at big events so that people have to take mass transit. It's for their own good, but they have to be shown the way.
Google has optimized for working 'most of the time' for 'the most people
This may be Google's goal, but it hasn't happened yet.
I don't have very many friends or acquaintances in the tech bubble, so I base my observations around real people in the real world. More and more they're giving up on Google entirely.
Their primary search engines these days seem to be Instagram, Pinterest, Etsy, Amazon, and other non-Google sources.
When I ask someone why they're searching Amazon reviews for tech support information, they tell me because it's not on the web. That's Google's failure.
You can only be thoughtful with your search if you know what you are searching for. But oftentimes i'm not really certain what i'm looking for, or i don't know the exact terminology that should be used, so i'll just enter some related terms, in the hope that google leads me in the right direction.
A truly thoughtful search requires an understanding of:
- What you're searching for, which as you mention means terminology and knowing that information exists. (If you don't know that there's a country called Burkina Faso, it's never going to occur to you to search for its capital)
- How each of your search tools works, its benefits and drawbacks. It's similar to selecting a programming language or framework: If I need to know a holiday date (e.g. I can never remember when the fuck President's Day is), I'll Google it because that's something even a normal person would notice if they screwed up. On the other hand, when I'm looking for current events information, I use a search tool that specializes in news searches for journalists and researchers because I don't want my search results biased by what Google thinks I want to see.
- The domain in which you're searching, so you can evaluate what the search tools provide for you and use the tools iteratively.
- Your own abilities and desires, which requires self-knowledge. A search is only a success if it produces something helpful to the searcher, and something they can't understand or won't use = not a successful search
- What information is and is not available. It sounds like a silly thing, but this is how a lot of scams work: They're testing for people who lack a certain subset of common knowledge. For example, I've seen articles talking about local elections that imply nefarious intent behind some information not being provided online, and they're obviously written by people who don't commonly work with local election data. Because if they did, they'd know that when working with local election data, the default is 'idk we have it in a file cabinet or on a computer somewhere'.
Search is HARD and Google has figured out one tiny, tiny part. It's just the part that was the easiest to build with what they had and that was easiest to monetize.
> Take away the junk food, and people will resort to real food.
Many people already resort to real food, even with plenty of junk food around.
“Problem” is unfortunately, that it comes at a price, that many are simply not ready or able to pay.
Who should step in is a good question, and probably governments should make access to information a right and have high quality public service available (in this case a public web search engine). Public libraries used to fulfill this role for centuries.
> Their primary search engines these days seem to be Instagram, Pinterest...
Why would someone want to search Pinterest? Every time I've gotten a search result to Pinterest it's been some scraped image completely and frustratingly devoid of the context I was originally searching for. Pinterest is one of the worst offenders on the web.
Another factor that isn't being fully accounted for is a new SEO/marketing technique where many people are asking scripted questions publicly on sites like reddit and then stealthily providing answers that market a product or service. This leads to reddit results not being exactly authentic as well. Pretty much most online reviews cannot be trusted as we are begged to do positive reviews of companies (and when companies outright purchase positive reviews, which is also very rampant) also as a factor.
Though Google is at fault for letting their service falter to the "payola" race, many other factors are in play all across the Internet since data quality has faltered almost totally. For major-cost and non-refundable purchases I need to trust, I go to brick and mortar stores and inspect what I am buying. I am thankful not everything has shifted to an online-only model. It's going to be a very bumpy ride on the Internet until Congress and consumer protection laws wake TF up and do their job.
Reddit does have astroturfing, but a lot of communities are aggressive about identifying and banning shills, so it's not as widespread as in google search results.
A ton of reddit communities are run by shills themselves. That's why users outright get downvoted and banned when they even are posting potentially winning content. We cannot act like everything is totally normal these days on Reddit. They're hemorrhaging users fast.
Just look up posts about people getting banned from reddit on Twitter or on black hat forums. There are tons of people complaining about it.
The thing about users getting entirely banned from a platform is that they aren't able to complain anywhere near to the forum about unfairness. It's all done in stealth.
Platform algorithms now often do a lot to keep dissent about platforms themselves and partner companies out of sight. It's pretty crazy how companies that publicly advocate equality and fairness serve to undermine speech when it's about their business.
There are such posts about each and every platform out there. In addition to not knowing what's the ban frequency, there's also no info in those complaints if they got banned for a good reason.
Last winter people got fed up with antivax-dewormer-nomasker bullshit and a lot of subreddits just took the trash out, so those complaints you saw might just be be those dumbasses.
I'm not saying there aren't annoying powertripping mods on Reddit, e.g. r/Linux, but there always tends to be an alternative subreddit one can use.
Is Google unique in that, though? Amazon reviews are worthless now because companies pay customers to leave positive reviews or pay review farms to leave positive reviews. Even if someone reports them to Amazon, though, the companies just close their accounts and open new ones with different names and sell the same product. It's so trivial for them to pivot when they get caught that I'm not sure there is a solution to this problem.
There is a perfectly good solution, and some people use it: don't buy from unknown vendors. If you buy a product from one of a few well-known companies in that space, they have a big investment in their brand, and are much less likely to engage in behaviors that might diminish the value of that brand.
The price is that you pay more -- effectively, you are paying for that branding.
I've had so many bad experiences on Amazon that I am increasingly doing just that. It doesn't apply to everything, but it is a useful strategy.
So how does that apply to the conversation around Google, though? We can't only ingest information from a small group of sources. That's antithetical to the entire concept of the internet.
Not really because the content is coming from a variety of users who are creating the content on the site. Reddit itself is not creating the content in the same way that sellers are trying to sell you their products and Google is trying to serve you their ads.
IMHO it's not so much of a problem with google search, but the internet as a whole.
Most genuine discussions have moved from open, publicly accessible web to places inaccessible to search engines and general public. Smaller niche forums, blogs and personal websites with no financial incentive have died out. People have moved to Facebook, Discord, Whatsapp, Instagram, Slack, Twitter and other places behind logins. Online newspapers and portals are increasingly using paywalls. Most of the genuine human interactions and quality content is not indexable anymore. Instead we have a million affiliate marketers fighting for the top positions in search results with every possible seo trick.
Reddit is one of the last places with huge amounts of publicly accessible online discussions.
It's because private communities are the only ones free from mass abuse. Public forums moved to private discord groups with hard to find invite links / etc because public forums take an army of anti abuse workers to keep alive. While a discord group just needs a few people to kick the trouble makers and maybe revoke the invite link for a while.
If that's the entire problem, then why not make these forums read-only by default (while still keeping them publicly viewable) and hand out invite links as in Discord?
Lobste.rs, for instance, has an interesting "invite tree" concept where your reputation is bound to people you've invited and who they've invited and so on.
But your overall point is valid, I think. My pointing to a single active forum doesn't change the fact that many of other enthusiast groups have moved to facebook groups and the like.
Feature, not bug. If there's no public search, it can't be gamed for money. The problem TFA identifies is one of discerning that the person you're getting your info from is an actual person, who cares. The best way of doing that, until we find some way of creating institutional trust in these matters, is talking to the sort of person that spends all day in talking to a chat room about whatever it is you're asking about.
I too have found myself searching more in Reddit. Not to throw shade on Reddit, but even if I find exactly what I’m looking for in there, it’s depressing that it’s all bound up inside of another walled garden who will eventually have the same incentive as Google: squeeze every last advertising dollar out of the produc… I mean users. Like Google, it’s just a matter of time before they too lose their balance.
A question worth posing to this community: how can we build an internet that’s hostile to advertisers? Secondarily, how can said internet also be much more accessible to content authors so they won’t have to learn a css, html, and JS to publish some stuff? Finally, how can that content be discovered from within this network?
I "search" reddit a lot, but all my searches are always through Google. Reddit search is notoriously bad, even after multiple attempts by them to fix it. Suffixing my Google searches with "reddit" though gives all the results I'm looking for.
A factor in there has got to be 'who pays?' If it's hostile to advertisers, then there's got to be money to pay for the infrastructure from somewhere.
Maybe a tax on ISPs? I think I'd happily pay $10 extra per month for access to an ad-free interrnet. Maybe $20. But how many of the people that are already happy with the ads and poor google results would do so? Would it be sustainable?
I think you're grossly underestimating how much is needed to replace the ad revenue that feeds today's Internet. Remember, majority of people wouldn't pay for getting rid of ads, because they don't have the disposable income. So they're also not really worthwhile to advertisers. You have to divide the revenue by some small fraction of current users.
Yep. It's really weird if you think about it. The first time I saw a company being upfront about their ad revenue [1] I was surprised.
> The Premium fee is basically about $7 per year, which is less than what a free user generates in ad revenue. Thus leagues that pay for Premium and use an ad-blocker are generating less revenue than free users.
They charge about $20 per user per year to remove ads. I pay for that, so I'm not sure what kind of ads they have, but I'd love to see what they're advertising and what the click through + conversion rates look like.
What you said makes sense to me. I wonder if advertisers are paying a fortune to acquire users with a lot of disposable income.
Agreed, Facebook makes something like 8 dollars a user per year? That's just one of the mega services, imagine replacing that money for all of the players that power your internet.
It's very relevant to consider that the value of users is very unequal.
IIRC for Facebook American users the average revenue was 50+ dollars per year. Furthermore, if you're the type of person for whom saving a few dollars doesn't matter, you're likely worth more than that average; and for FB to break even on your ads an appropriate price is likely to be closer to $100 per year.
There must be a difference between paying for the infrastructure and paying for the content (including the code).
It's easy to imagine a future where the infrastructure is something analogous to a public good, like clean water. The infrastructure is agnostic to the content.
The content, on the other hand, is a product of a massive, churning, never-ending process of intellectual work. Ads, for the most part, provide the underlying economic incentive for that work.
And the dirty little secret of all that ad-supported creative content? Its value is minimal. Truly minimal. Often low quality, incoherent. If it all went away tomorrow, I would hardly notice.
That it is nevertheless so popular is something of a paradox. I know I sound ridiculous, but at the core, I'm beginning to believe that the fact that advertising is as powerful an economic force as it is, is a reflection of the fact that people's lives are more boring and devoid of meaning than we often admit.
Ads will always exist because they work. Only way is to ban them explicitly but you can do that with AdBlock for example but you still get SEO spam, placed content, inauthentic "recommendations".
> how can we build an internet that’s hostile to advertisers?
You have to reify "trust" into concrete, computer-representable data. Maybe borrow the "web of trust" concept from PGP, but do some sort of multiplicative thing where the amount you trust someone's recommendation online is the product of the trust relationships between you and the recommender. That's really the best you can do - even legislation against online advertising will be subverted by companies that go through layers of proxies to buy influence.
One small thought — having the search engine be configurable, so that the user can specify which sources to give priority to (e.g., Reddit, NYT Wirecutter, Wikipedia, etc.), would be an incremental improvement.
I, too, search "<search term> + reddit" often for product reviews and such. Thing is, the results on that front have started to slide as the paid review side of the internet catches on. I'm finding that it's getting harder and harder to trust the reddit search results - lots of shill accounts and obvious junk. That's not a google problem, specifically, but it's another degradation of a workaround for declining search result quality :(
Usually I trust Reddit threads where users give pros/cons of multiple competing products. Things like running shoes users have usually tried a lot out and liked them for different reasons. If those match with the one pair I've tried out it seems like a useful data point for decision making.
Yeah, a lot of subreddits are clogged with the same bad info that's gotten all over Google's front page. The stickied "list of recommendations" on an enthusiast sub is just the same as you'd get from clicking the top result of "Best X 2022" on Google, complete with affiliate links
That's not the point - the point is that paid shills are astroturfing reddit enthusiast subreddits, so that operator doesn't shield you at all. Once a hobbyist subreddit gets big enough, it attracts a lot of attention from shady types trying to capitalize on the captive audience. I've seen it happen numerous times (only when the offender is caught) in the /r/watches subreddit, the /r/overlanding subreddit, etc.
About the "dead internet conspiracy" - I've worked in writing how-to articles for a fairly large "help" website. They paid very little attention to the quality of the articles. I was paid for each piece and thus had about 30 minutes to write an article and later integrate feedback from internal review. Otherwise the payment became too low.
The most important factor was cramming SEO terms and links to keep people on the website into the articles.
The result is trashy articles that could well have been written by a bot but aren't. This could possibly be done with the help of curated bot-content, but I think we're far away from the point where this is really more profitable than getting students to do the work.
>> This could possibly be done with the help of curated bot-content, but I think we're far away from the point where this is really more profitable than getting students to do the work.
It may be becoming borderline. I expect that sentence/paragraph completion is already becoming useful to people who churn out quick content for a living. In any case, the important part isn't whether or not it's bots. The important part is whether or not it's authentic. The precise meaning of authenticity gets squishy, but it exists nonetheless.
IMO the sentiments are correct, whatever the details. Part of why google sucks is that the internet is worse, for a bunch of the things we use google to search for. The internet becoming a larger, more profitable industry changed it. Instagramming for influencer perks, SEOing, or selling targeted ads like FB do... it does not lead to the same places that earlier iterations of the WWW produced. Times change.
My friend briefly had a copywriting job writing weed strain descriptions for dispensaries. He was never provided the product he was describing, just told to make it up.
I have the very same feeling concerning electronics. Searching for a particular product does not even popping up 5-10 comparison articles but the content of all seems to be based on technical specifications of the manufacturer only, which I already have a hands on.
Significantly more time required for consciously choosing a product to purchase (which in my case is critical because I am like Sheldon Cooper trying to choose between PS4 and XBOX One normally, to the horror of my wife, she wants a new TV and it is months long project based on accurate and quick data, and now this, with Google, which makes our family atmosphere even more tense : ) )
In case you haven't encountered it yet: https://www.rtings.com is a good site for TV specs/reviews specifically. I know someone who works there and their methodology seems legit (I like it more than Wirecutter).
Most reviews are useless from the start because most reviewers are totally dependent on manufacturers or dealers providing samples (yet will generally claim to be "totally independent"). That's before you get to reviewers who can't or don't know how to test the product in question and so end up narrating the manufacturers specs to their faux testing.
Niche, probably outdated, but indicative: The only way to purchase a laser printer with high printing quality these days is to buy something expensive and hope for the best. Magazines used to do actual reviews of these things.
Brother MFC laser printers have delivered for me for over 15 years now. I have bought many for various small offices and my home and relatives’ homes, etc, and I have not heard any complaints.
I especially like the scan function where brother web connect OCR’s the document and saves it as a pdf directly in your Box/Dropbox/OneDrive/GoogleDrive folder. Just wish it worked with iCloud Drive.
Oops, my bad, I did not properly take into account high quality printing. It is definitely too cheap to be anything high quality, they just get the job done for everyday printing at a low price.
I look back at the newspaper stories I wrote a few decades back. I could get the score from the coach, find out hits from who and when, and after that 20 second interaction, I could write a news story in maybe three minutes, which told you everything you needed to know.
Somebody who worked at a winery once told me that the flavors they mention on the bottle are actually what the wine is missing, and they name them in the hope that the power of suggestion give you a more balanced impression.
This is something I notice in advertising generally: whatever they most emphasize is least likely to be true about the product (e.g., the "great taste" of McDonald's).
It would be interesting to see some sort of study, to see the impact on wine labels on: normal people, vs wine "experts", vs sommeliers.
I've seen a few of those wine documentaries about sommeliers, and they certainly made it seem like it was a legitimate ability to identify stuff. I'd be interested to see how close they are in a more neutral, measured environment.
>Tasters can't even tell reds from whites with a great deal of accuracy. [2]
Actually, this one is bullshit. While it may be true that the average person without blind tasting experience cannot do this, anyone who has actually done blind tasting seriously should be pretty accurate at this. Purely structurally, red wines generally have much more tannin and more alcohol than whites.
Sommeliers are professionalized, with courses & exams. But I don't believe they are actually judged on their ability to taste wines and detect flavours. Designing such tests would be simple - rate of successful flavour identification based on data from other Somemeliers.
Instead Sommelier exams are subjective - candidates are judged by another Sommelier on subjective criteria rather than objective measurement. In my opinion they judge it as a dramatic performance: how quickly can the candidate rattle-off various flavours? How "high class" is the language they use to describe the flavours? How does the candidate present themselves? Sommeliers dress in fine suits, but this should have no impact on actual wine tasting ability. Yet I am sure if I showed up in a draggy old t-shirt I would fail regardless.
The whole industry seems allergic to objective scientific measurement. I am sure Sommelier's do have some ability to identify varieties, but are they actually tasting all those subtle hints they list off? I doubt it.
What is this comment based on? To me it feels like this is what you imagine the exams are like. The MS exam (the highest and most famous one) has three parts: theory, service, and tasting. The service part is all about presentation, absolutely (that's the whole point). The tasting part is not. In particular...
>But I don't believe they are actually judged on their ability to taste wines and detect flavours. Designing such tests would be simple - rate of successful flavour identification based on data from other Somemeliers.
That is exactly how the tasting portion of the exam works. And
>In my opinion they judge it as a dramatic performance: how quickly can the candidate rattle-off various flavours? How "high class" is the language they use to describe the flavours?
This isn't really how it works, especially the "high class" language comment. Actually there are a variety of different things that they are supposed to identify about the wine, known as "the grid" [1]. You can take a look at it yourself. They are essentially judged based on how well their grid matches up with the consensus grid from the master sommeliers.
The other day I looked up the wordle answer (I know I know). The first result was a site where I had to scroll through about 19 paragraphs of SEO vomit to get to the answer. The page could literally have one word on it and serve it's purpose. If that isn't a sign that the the internet, or at least google search, is dead, I don't know what is.
There's a whole industry of sites like this for NYT crossword answers, to the point that it's often impossible to get any organic results at all for something that's been clued in the Times, which is frustrating because I don't just want the answer (there's a button in the app that does that already), I want to learn about the thing I’m unfamiliar with. Luckily, most topics that come up have some content on Wikipedia so I go there instead.
Ugh yes that’s super annoying! I don’t want the answer spoon-fed to me. The point of searching it is to actually learn a little something which will help me remember in the future
lol the same happens with WSJ. It’s very disappointing to try and do a bit of research on a clue and get the answer on a bot site linked to the puzzle you’re doing. Interesting how they do it though.
It’s like a DDoS attack on your mind (distributed because everyone is doing it). The attention span economy at its finest.
Reminds me of those ways to catch spammers or bots by occupying some of their resources with meaningless tasks for as long as possible. Except it’s turned around.
Sometimes I wonder if there’s even any real money in ads anymore or if it’s just a giant circle jerk that slowly destroys society…
Speaking of bots, I'd be interested to know the percentage of articles on major traffic content sites are authored or co-authored by AI.
My suspicion is this is rife given how many articles read poorly and are almost entirely fluff. If this is true it would appear we are doomed to algorithms shaping our online experiences, which is worrying given the existing shrinking diversity of opinion and content. It's like a entropic gene pool in nature, but with information.
Sports stories are frequently written by bots/AI/whatever you want to call them. Sports stats makes it relatively easy to create written text articles about games. When first rolled out, there were some obvious tells such as an excessive amount of "for the first time this season" or "set a record for the season" in articles about first games of the season. Unusual plays or quirky behavior by participants tend to be missed in these articles but otherwise they're serviceable though somewhat dry and bland.
Stock analysis articles are very commonly generated automatically from templates. The selected template is based on some real characteristics of a described company (whether the stock went up or down recently, what is the P/E etc.), but the content is generic and reused across all companies with similar characteristics.
Apparently, using a machine translation as a basis and working through correcting it to read or write foreign languages is a growing trend and that’s a form of computer assisted literacy if I were to guess.
Bar that, it’s humans outrunning AI in the race to the bottom with a head start. Human people can be forced to be incredibly machine like.
> Human people can be forced to be incredibly machine like.
Amazon has a site for humans to pick up small tasks and get paid for them called mechanical turk [1], which is a reference to a fake chess playing machine with a human inside [2]. With the great resignation and the workforce otherwise pushing for a reasonable standard of living, I'm not sure how heavily mturk is still used as depending on tasks and speed it's really sub-minimum wage work for many people. [3] But as The Atlantic article says, sometimes it's the only work people can get.
I think the quality of searches like “best TV” will improve dramatically once language models are used to generate SEO spam. Anything would be an improvement over what today’s human spam bots produce.
Lately, half the results I get are pages that I cant view without paying to some service or signing up for a free trial.. Google literally serves up results that are unreachable ..
Can we also talk about how Google allows top organic results to LinkedIn, Instagram, and Facebook which literally present you with a LOGIN page before you see anything related to your search?
Absolute worst UX ever, yet they allow it because it's their SV buddies.
> yet they allow it because it's their SV buddies.
At one time, Google used to ban sites for "cloaking": offering one version of the page to the crawler, and another to the user.
But over time they got into trouble from these sites, getting sued left and right. Being accused of putting up a wall, and abusing their monopoly. Eventually, these sites won out and Google dropped this requirement.
Similar story with Image search. "Blah blah something something copyright something". And voila - now you can't get direct access to images on the search results page.
I mean then is DIT really a conspiracy theory? I know that people in HN have already started doing replies with AI in some threads. (only because they tell people). Wait a few more years and there will be no way to know if forums are just bots generating content.
I guess HN would be a weird outlier, because there are no ads. Except maybe the bots would be useful after all when various "Show HN" or "Launch HN" and have the bots cheer those companies and get random publicity.
The idea that a genuinely Dead Internet might be an improvement over the current internet experience is a fun one, and one I don't entirely disagree with.
Within the last 6 months I’m seeing a rise of very good NLP content farms that almost have what I’m looking for but not quite. I think what they do is start off a transformer based NLP with example queries that others have searched for and it generates a realistic looking answer that’s usually wrong. The scale and breadth of these could only be done by machines.
Writing SEO content isn't necessarily one of them anymore. Check out AI content services like https://www.frase.io/. You can also generate things like product descriptions inside the GPT-3 playground.
Reminds me of an old The Parking Lot Is Full comic.
> Little-known Fact #839: There are only twenty-three people alive today, and you're one of them; everyone else you know just looks human to lull vou into not searching for the other twenty-two. Lonely? You should be.
I would expect an automaton to be able to spell; so perhaps the presence of spelling errors is a mark of an authentic page. Maybe one could force Goo to spit out authentic results by including a strategically-misspelled word in the search terms.
Classic example of this kind of content... Try searching "How to use X to get stains out of Y".
You will find a page for almost any X and Y combination. And they will all have wording like "Put some X on the stained Y... wait a bit... rub it in... and then put it through the washing machine. Hope it works!".
Totally agree on the reddit point, I've also noticed the same occurring to me. The girlfriend recently got Pokemon Arceus and sometimes asks me to Google something she wants to know.
It's completely pointless, you just get a bunch of articles from news sites (??) that transcript the quest but not tell you anything more. I miss a nice community wiki like I'm used to from playing Dark Souls etc.
I've just started appending site:reddit.com to everything, works a lot better.
Yeah, Fextralife saved my ass multiple times while working through the dark souls series. It's a shame that type of community resource isn't more popular.
Google could fix this by making the algorithm take into account searches that often end with "reddit", thus applying more weight to Reddit results to similar searches where the user didn't include Reddit in it. Clearly it's an indicator that those are the better results.
Take StackOverflow for example. Almost any programmer will find a SO result as the top result and it's usually exactly what you're looking for. Since there's no money to be made by companies writing blog posts on debugging a compiler error, Google's algorithm works as intended.
Question is: Why hasn't Google done anything about this?
It's the organic results that are terrible, so they're not losing ad revenue by placing these garbage sites at the top. Perhaps its to intentionally make better websites pay for ads to get better placement? But those won't be the ones to ever pay to begin with...
I think it would be foolish to assume google hasn't spent hundreds of hours in meetings talking about what they can do about everyone having to type reddit. Problem is they are facing an army of SEO experts who are one step ahead of google. As well as legal issues. Imagine if it was found google was artificially boosting reddit in an unfair way.
I agree google is bad, but I think reddit is rapidly becoming equally as inauthentic. I'm sure every major player at this point understands the gains that can be had by astroturfing reddit. The real problem seems to be the internet is inherently untrustworthy and going back to finding people you trust in the real world is the only fix I can see.
We've also got to address the people and corporations that are gaming the system that google has created. Google is by no means off the hook, but Marketing practices have also taken a very bad turn to deception and in reinforcing a payola systems recently that we may never be able to recover from trust-wise.
I did software dev at a marketing firm for about a year, and it was pretty soul sucking, so I know what you mean. I won't work at one again unless it's literally my only option.
I’ve been there too. The sad thing was that the people I worked with were genuinely great and smart people but you’re spot on about the actual state of the environment being a depressing and soul draining thing. I quit that job on moral grounds, I just couldn’t be a part of that any more.
> - "Why are people searching Reddit specifically? The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust."
Haha, the noobs. I use HN instead sunglasses cool face
A bit more seriously: I fully agree with this. And if HN doesn't have what I'm looking for then I use Reddit as well. But if HN has some info on the topic with a few highly upvoted threads, damn, it always impresses me.
When I'm looking into some project or piece of software I'm unfamiliar with, I really do search for HN posts on it. Fastest way to cut through (enough) of the biased material and get something genuine. Even hyped stuff usually has enough contrarian posts to give you an idea of where to look for the skeletons.
HN has now been around a while, is really popular, and is starting to catch the attention of ad/marketing companies now, though - how do you know what here is genuine?
Even the heuristic of "only trust accounts older than n years" isn't perfect, as eventually a few people will undoubtedly sell their old accounts on a dark web market for a little extra cash...
And it's not just the spammers. Any topic that touches domestic or international politics in any way almost instantly brings out a lot of bad-faith actors, here or anywhere else on the internet.
> Even the heuristic of "only trust accounts older than n years" isn't perfect, as eventually a few people will undoubtedly sell their old accounts on a dark web market for a little extra cash...
Yikes, I hadn't thought of that angle. That would explain some of the long-dormant accounts on Some Other Place that suddenly start spewing out-of-character garbage, assuming they weren't password-guessed, data-breached, or keylogged by some rando.
Hmm, I think for now it's relatively easily distinguishable. I've noticed that HN'ers feel "like me". So I look for that specific signal. I look for signals that care about curiosity and an insane hunger for the truth. Also, it's tough to mimic "Oh yea, I was at Xerox Parc programming this language that was posted about, let me give some nuanced insights into what was stated in the article" (that quote was made up for the sake of example).
The only marketeer that would be able to consistently fool me might be a marketeer that was a developer. But last time I applied for a junior marketeer position I was asked: given your resume, don't you want to be a developer instead? So devs seem to get pidgeonholed into always being devs.
Me too I even have a bookmarklet in Firefox so that I can use a prefix (hn) and the search is rewritten as site:news.ycombinator.com, to make sure all results are limited to HN.
I also have the same kind of bookmarklet for Reddit and Google Scholar.
The reason that the quality on Reddit is higher is because there’s people moderating those quality subreddits. Without those moderators it would all turn to crap and be just as useless as Google.
(This isn't an argument against your point, just a bit of additional context that increasingly is odd to me as Reddit gains more and more social weight)
Very true. Fortunately I don't think most moderators want to see their community destroyed. That's exactly what would happen if they started taking money. As a group, people will catch on fast and either force the moderator to step down or just leave for a new subreddit.
> - "There’s a fun conspiracy theory that popped up recently called the Dead Internet Theory..."
> I hadn't heard of this. Now that's some sci-fi level of conspiracy but in today's world it seems totally plausible.
I never believed in conspiracy theories, and after I read "Media Control" by Noam Chomsky I understood there is no need for conspiracy theories once you understand how individual incentives are aligned and how individuals always act to maximise profits.
Someone on HN phrased this and I am not taking credit for it but it explains beautifully whats going on: "Google is not making money by showing you the best search result they can, they make money by keeping you searching."
That does not make sense. If searches did not result in satisfactory results, then people would stop searching.
Which they are, evidenced by restricting searches to HN or Reddit.
This is a problem for google, maybe not right this minute as growth might offset dissuaded users, but nevertheless, it does not behoove them in the long run to provide garbage search results to people.
> Which they are, evidenced by restricting searches to HN or Reddit.
You are right but overestimate the number of people doing these restrictive searches. They should be in the <0.1% range. I personally switched (to duck and latly kagi.com) and know quite some people in tech that did so too, but the non tech-savy person (which are the majority of overall users) doesn't even know other search engines.
Eventually, though, even non technical users will stop Google searching if it wastes their time. If they do not find what they want, then they can go back to asking people in person or instagram or whatever, but I would not expect people to keep aimlessly searching.
The death of the authentic web really chimes with me. I have an almost physical reaction when I occasionally come across a page that isn’t trying to sell me something, that is a labour of love.
A month or so ago, I was trying to help someone retrieve some very old Wordpress for Mac files. I found http://www.columbia.edu/~em36/wpdos and was so touched, I sent the author a few dollars for a coffee
Google's problem is they've virtually nothing (given their resources) to "commoditizing their complement". https://www.gwern.net/Complement
Google's compliment is web sites. What have they done to make a web site easier to make?
They even killed their RSS feed. They have released a bit of web tech, but their offerings are generally a bit sad or only solve Google problems (e.g. Go).
If you want to distribute an .exe or .app, MS and Apple have released some pretty good tools to help. If you want to write a blog or make a simple web app, it's unlikely you're going to think "Google has some great stuff to help, and has awesome tools". Mozilla's web resources are better. Microsoft's web resources are better.
>> "Google increasingly does not give you the results for what you typed in. It tries to be “smart” and figure out what you “really meant" ..."
> This is the most annoying behavior because I really mean what I write.
Yeah I remember this being mentioned in a local presentation at university. As a great thing. Google doesn't search for what you write, but what you want.
The problem is that very often Google don't know what I want. Before they introduced this, I was able to define my query so that I got exactly what I wanted.
I think the OP is weak because it conflates ads and seo spam. Yes, Google went all in with ads, and yes, this hurt its credibility and the quality of its products.
But there is no conceivable universe where seo spam isn't the arch enemy of Google. Google needs to fight spam to survive, it knows it and it does. But that's hard. So hard in fact, that nobody else has cracked the problem, and for all the anecdotal evidence of someone switching to Bing, Google's marketshare is still utterly dominant.
The Dead Internet Theory may have some weight: Google hasn't dropped the ball, but it is slowly drowning into the sea of "content-free content".
That said, the reason the Reddit trick works is that it uses information Google explicitly excludes when ranking content (engagement signals).
Google has a bunch of “objective standards” that it uses to paternalistically shape what the web looks like. Many of these are divorced from what users actually want for pieces of content (https, AMP, a life story in front of recipes to demonstrate authorship, etc).
As explained in another comment [0], the real reason the Reddit trick works is that Reddit content is hand-curated by humans who are dedicated and passionate (and, I think, also work for free). It's a garden in the jungle.
That inauthenticity comment really hit home for me, too. I realize that I do not trust the internet at large, and haven't for a long time. That's been the real trigger for my retreat from mass social media into smaller, tigher online communities.
Even HN is starting to feel like it wants to sell me something.
The latter feeling may be because HN is run by a startup accelerator. They run literal native ads on the front page whenna startup they sponsor goes live.
I don't think that's where it comes from. It comes from the same places as Google, where people know the value of this community's attention. Posts at the top of HN are designed to get to the top of HN.
This fits well with my own worldview. I've been griping about Google results for years, and jumped for DuckDuckGo when it became usable. I'm sure that fifteen years from now, DuckDuckGo will be ad-infested crap and someone new will come along to replace it, just as Google replaced AltaVista.
Even DDG knows that it can't handle everything, and so it has its bang shortcuts. I've used the !reddit one, and I'd use !w (Wikipedia) except I do those from the Firefox search bar.
I've heard the "everything's a bot" theory before, but never saw a name put to it before. I'd have to guess that 99% of all SMTP traffic is spam at this point.
>I'm sure that fifteen years from now, DuckDuckGo will be ad-infested crap
In terms of direct ads, perhaps. But for SEO spam, in many cases, DDG already seems to be there. For example, things as simple as "python datetime", "python json", or "python datetime.now", where it would seem obvious that the top result would be the documentation for the module/function, have spam sites above the actual Python documentation. Meanwhile, search for "matplotlib", and your screen will fill up with ads.
> This is it for me exactly. I search for the following kinds of things on Reddit exactly because results on other sites aren't trustworthy: Reviews are secretly paid ads.
So are reddit-comments and entries. There it's even worse, because most people don't connect the content with manipulation.
> The "best" recipe for pancakes is only what's trending on instagram right now.
So, like reddit? I mean every platform has their hive mind, and reddit is even worse, because the hive mind can be manipulated with paid upvotes, not just reposting and comments.
> The same for trending programmer tools.
Aren't most of them yet again commercial products from companies?
"It is obvious that serving ads creates misaligned incentives for search engines..."
I don’t think this is the problem. The problem is the need for public companies to grow exponentially. If you take away the need to constantly grow exponentially, then ads on search can be both balanced well and make an insane amount of money.
There’s something sad and ironic about using Google to search Reddit. One, I mostly dislike using Reddit - I only want to see specific discussions very occasionally. Two, what is the state of the internet if I have to use the best search engine to find content on a website I mostly dislike? Haha.
Reddit's search engine is kind of crap (not terrible, but also not great). That's why I use Google for Reddit, to have a better Reddit search experience.
I figured that's why it's so high, is Reddit's UX keeps slowly getting worse so the best way to find stuff on Reddit is by searching outside of it.
>most of the web has become too inauthentic to trust
i think that specialised search engines are gaining ground. For example, I am using github search for searching code samples, that works better than google.
You might want to check my side project that tries to explore the subject. I have a search tool / catalog of duckduckgo !bang operators, i am hoping that it allows for better discoverability of specialized search engines.
The latest addition is a description for each search engine, just hover over the name, and you get a description derived from the sites meta and title tags.
I think that specialised search engines are gaining ground, it has become easier to set one up, thanks to elasticsearch/lucene. They can be quite good, for a limited domain, and they don't have to invade your privacy in order to find out what you are looking for. I think that what is missing are tools like this, that would aid the discovery and use of these search engines. I hope that this will allow them to eat into the market from the 'low end'.
>because results on other sites aren't trustworthy
Reddit is gamed way more than google. Paid posters, moderation of anything against a narrative. Google search may be dying, but reddit ain't doing much better.
> The latest conditions on mountain bike and hiking trails are being shared inside communities like Reddit but not on the web
This point especially rings true for me, but it also concerns me a bit. Reddit has killed a lot of other forums over the years. If something happens to Reddit, we run the risk of losing a large corpus of information.
> The long answer is that most of the web has become too inauthentic to trust.
It's only a matter of time before reddit too becomes too inauthentic to trust. Not only is it directly funded by advertising, its audience is mainstream enough for advertisers to invest time and money posting fake opinions in order to make it look like it's coming from real people.
I seriously hope I never see comments or news about people appending hacker news to searches. I don't want advertisers to kill this site when they catch wind of it.
I agree. Often, what I am after online is to see what other realpeople are saying about something. typing 'reddit' into google is basically a proxy for "please google for the love of good, can you start indexing actual human discussions again?".
I have a similar habit, I often search "[topic] forum" - I'm not fond of Reddit specifically due to accessibility issues although in some cases I still go there because it's the only good source.
Searching Reddit helps but the quality of comments has gotten lower since 2015 or so. It seems to coincide with the wave of subreddit bans and the nakedly politically-driven moderation on subreddits. And with the reflexive attitude—against anything countering the Reddit consensus—that developed during the Trump years. High quality posters seem to have withdrawn from the site (at least in how much they comment) and what's left is mostly ignorant teenagers and bitter millennials with shitty jobs. In turn, that crowd is much less likely to upvote high-quality thoughtful content, so the cycle continues. The decline in quality has trickled even into the less popular subs. Don't get me wrong, the site has always had problems, but the more recent decline in thoughtfulness is dramatic.
The worst part is that despite Reddit getting so much worse, there is no other place that's grown to fill the void. This place is great, and I do search HN when it makes sense, but it's small and narrow in scope. Reddit basically crowds out any competing websites by sucking up all the low-level chatter required to sustain a community, but has also pushed away high-quality posters, who now have no place to go. Very tragic but maybe a good case study in shitty network effects.
Quality on any non niche popular subreddits were already abysmal long before 2015. Reddit is useless for anything which is not highly specific but there are some diamonds in the rough: great subreddits exists about fashion, knives, gardening, coffee, shaving and plenty of other weird interests.
This is true but I find that about 2015 is the inflection point. Even posts about highly technical subjects are not as good if they’re from after that.
I dunno, looking at the growth curve in Paul Graham’s tweet I expect most of the drop in average comment quality can be attributed to the size of the user base. It’s hard to keep high-quality content the norm even in much smaller communities.
/r/nfl had a reputation for high-quality content and wasn’t a particular battleground in the Trump Wars. It’s still a good breaking news feed, and the live game threads are fun, but every post is dominated by joke comments and memes.
I find the idea that banning Trump supporters killed reddit to be pretty far-fetched.
To the extent there is a change in quality, it probably comes from other factors, including having a bigger, broader, and different user base now than in the past (and only a small portion of that change likely came from Trump-related bans).
Parent didn't say Trump supporters. And I completely agree with their observation. Quality went completely off a cliff around that time. I think it just officially entrenched the Reddit orthodoxy and that there is a "right way" to think, and that infected even non-political subreddits. Not all though, I know some subs are much better and open to discussion than in the past.
It's perhaps a little bit early in its creation to be sharing this but I am working on a new search that should help to fix the problems mentioned in the article, https://namusearch.com/. It allows you to build (and share with others) a curated list of websites that you want to use for searches
Yup, when I want to search anything I use a combination of reddit, HN, and Discord. My main use of Google these days is to find a website I forget the name of but roughly know what it's called. In the olden days, I used bookmark aggregation sites like del.icio.us to search for relevant content, which was generally more fruitful than a Google search.
> I search for the following kinds of things on Reddit exactly because results on other sites aren't trustworthy: Reviews are secretly paid ads.
There is so much shilling on Reddit if you knew it would blow your mind. I wish more people realized this. Reddit is the best place to shill because not only is it ridiculously simple, people also automatically assume you’re not shilling, and then once you seed the idea, everyone else will do the shilling for you indirectly.
The healthiest way to use Reddit is like Wikipedia: assume the information you’re reading is highly compromised and biased in one way or another, but use it as a starting point in your further research and it’s a great tool.
Reddit posts are not your friends. Upvotes do not mean the contents of the posts are legitimate or not shilling.
Reddit is the best place to shill and the sooner the non-shillers figure that out, the better off the entire internet will be.
Reddit also - in my opinon - actively enables shilling and botposting. Why do they have an API?
A forum that's meant to be 100% about humans talking to humans doesn't need an API, so why does it expose one?
Also the model of user-created and user-moderated subreddits actively enables the creation of shill accounts. It's trivial to create a subreddit and use it to farm karma with a ton of bots. If you can keep real users from ever entering your walled garden of a subreddit (of which there are many) your bots will never be detected until you wipe their comment history and set them loose on the rest of the site.
I didn’t really even think about this properly until just now.. these days I am looking at Reddit, Facebook groups and if needs be, YouTube (videos not by ‘creators’ as far as possible) to find information I used to google. Ads and referral links have totally ruined the usefulness of so much information.
I agree that Reddit remains a good source for info, but I’ve found that Google usually does a good job surfacing Reddit results — usually in the top half. Though this is most notable when I’m googling for esoteric info about TV shows and video games (e.g. “best build mass effect 3”)
I view the Dead Internet Theory as Black Mirror style satire. All it would take is liberal application of GPT-3 style transformer AI to content generation and much of the Internet could be fake. You could have fake political trolls arguing with other fake political trolls from the other side, fake blogs, fake review sites, etc. and it would take me a while to notice. Most of the modern Internet is just that bad.
Advertising always creates perverse incentives. It works in traditional media too. Look at what happened to things like Discovery and The Learning Channel when they became subject to advertising based pressure for ratings. They went from having actual educational content to being full of tabloid trash.
"The latest conditions on mountain bike and hiking trails are being shared inside communities like Reddit but not on the web."
I just wanted to mention that a friend of mine made an app for user-reported trail conditions that might be worth taking a look at:
https://trekko.app/
Dead Internet Theory is totally believable. I remember back in 2006-7 or so, being slightly curious about putting up a food review site because I was really angry at my local X food establishment. I found places were you could buy complete restaurant database ready to be scripted onto the web for maybe 90 a pop. The data was actually pretty good, but I was shocked by the huge community around flipping these dbs into internet spam. For the very high majority these were hungry business types who could barely open a code editor without asking for help. I only expect the problem has gotten exponentially worse since then now that ai generated content has improved in quality.
I think there’s actually two dimensions to this problem and the article and your comment only address one of them.
The other dimension is that, in the past, if you searched for stuff your results were likely to be a blog or a forum thread. Today the bloggers have evolved into instagrammers, TikTokkers, YouTubers, or Podcasters. The Forums and community pages have moved to Facebook, Twitter, Slack, Discord, etc.
So it’s not just that SEO and botspam has eaten the Google results page, it’s that this is all that’s left of the open internet that needed search to navigate. Much of it truly is a wasteland. Google owns part of the blame for crippling RSS and privileging recent pages and specific domains or AMP pages over evergreen, self-hosted content in search results. But also users have given up on the open internet in droves. Instead of starting a fansite they start fan subreddits or discords instead.
Based on some of the April Fools' Day experiments that Reddit has done in the past, I'm not sure why you wouldn't have the same hesitation and mistrust of Reddit posts and comments. So much of the content, even on Reddit, is made by bots or copied by bots from older, legitimate user-generated content.
They have filed for an IPO last month with the SEC so they should go public very soon.
Last valuation was at 10b$ which is ridiculous for a website that can literally get its most popular subreddits shutdown arbitrarily whenever a small group of extremely online volunteer mods decide to "go on strike" by locking the subs because they don't like something/someone else on the website.
It happened before and the admins yielded to them so I don't see why it wouldn't happen again, especially since it's not like they can run the website without that weird cabal of (mostly delusional/psychotic) power mods doing their work for free.
>>is ridiculous for a website that can literally get its most popular subreddits shutdown arbitrarily whenever a small group of extremely online volunteer mods decide to "go on strike" by locking the subs because they don't like something/someone else on the website
I have a story about this - and its worse than just mods -- Admins intervene and set narrative on for whom is allowed to mod and make mod decisions...
I wont reveal the details - but I have seen Admins literally come in and fuck up Mod orders because (my suspicion) is that the Admins have MANY accounts that /appear/ as mods - but are actually Admin shill accounts...
I had this happen to me first hand and I was appalled.
Reddit's ethics are absolute garbage in this regard.
I've also thought that. The reddit shutdown made the frontpage of (reputable) news sources, I can't imagine that investors aren't going to be asking the admins how much control they really have over the user experience on any given day.
regardless, what is reddit's current value? (I've been on reddit for 15 years - but I have recently deleted my accounts due to censorship and ban-hammering for the most ridiculous reasons.)
>> "Google increasingly does not give you the results for what you typed in. It tries to be “smart” and figure out what you “really meant" ..."
> This is the most annoying behavior because I really mean what I write.
I hate this too. I do get typos corrected by Google. But I don't need that - if I put a typo in my query and get bad results, I can correct the typo myself. But if Google decides I must not have meant what I actually said, there's no way for me to correct that. It's a ridiculously bad tradeoff - we eliminate errors that are trivially fixed by introducing errors that can't be fixed at all.
I have similar feelings about phone input autocorrect, which automatically converts typos that are very easy to read into (mostly) correctly spelled words, plus (sometimes) completely unintelligible nonsense.
Its not about reddit as a search, it's about using reddit to validate your search because the alternative would likely yield poor results. You could trust the 10 listacles that came up as the first results that all look oddly similar, or you can try and filter through reddit by including it in your search terms
- "Why are people searching Reddit specifically? The short answer is that Google search results are clearly dying. The long answer is that most of the web has become too inauthentic to trust."
This is it for me exactly. I search for the following kinds of things on Reddit exactly because results on other sites aren't trustworthy: Reviews are secretly paid ads. The "best" recipe for pancakes is only what's trending on instagram right now. The latest conditions on mountain bike and hiking trails are being shared inside communities like Reddit but not on the web. The same for trending programmer tools.
- "It is obvious that serving ads creates misaligned incentives for search engines..."
What I'm shocked by is that Google somehow maintained a balance on this for so long. Well, at least a good enough balance that people still use it primarily.
- "Google increasingly does not give you the results for what you typed in. It tries to be “smart” and figure out what you “really meant" ..."
This is the most annoying behavior because I really mean what I write.
- "There’s a fun conspiracy theory that popped up recently called the Dead Internet Theory..."
I hadn't heard of this. Now that's some sci-fi level of conspiracy but in today's world it seems totally plausible.
reply