Good Search Engine Bad Search Engine

There are things the major search engines (Ask, Google, Microsoft, and Yahoo!) do that are good and things they do that are bad. “Good” means they benefit the Searchable Web Ecosystem. “Bad” means they do NOT benefit the Searchable Web Ecosystem.

Here is a brief rundown of some of the best initiatives and worst abuses of the major search engines (based on behaviors I have observed over the past year).

Bad Search Engine – Abusive Practices That Hurt The Searchable Web Ecosystem

• Imply that the United States Government approves of and supports your guidelines. (GOOGLE)
• “Test” servers by making up fake URLs and crawling them. (YAHOO!, MICROSOFT)
• Artificially promote Web sites to the top of your search results. (ASK, GOOGLE)
• Focus on other search engines instead of your own. (MICROSOFT)
• Publicly humiliate people for not abiding by your guidelines. (GOOGLE)
• Report bogus data to Webmasters. (GOOGLE, YAHOO!)
• Change your service without explaining the change. (all four)
• Refuse to publish (even supportive) comments on your blog. (ASK)
• Automatically nofollow new user-generated content on your site. (GOOGLE)

Good Search Engine – Things You Do That Help The Searchable Web Ecosystem

• Share site-level information with Webmasters. (GOOGLE, MICROSOFT, YAHOO!)
• Announce new features to your visitors. (All four)
• Blog often, blog broadly. (GOOGLE)
• Offer secondary search capabilities (deeper, configurable site search; custom search). (GOOGLE, YAHOO!)
• Answer questions directly on your site. (GOOGLE)
• Automatically nofollow new user-generated content on your site. (GOOGLE)
• Engage with Webmasters through multiple conference circuits. (All four?)
• Use your own resources to promote yourselves. (ASK, GOOGLE, MICROSOFT, YAHOO!)

Neither list is exhaustive by any means. I placed the bad stuff first for two reasons: Search engines are abusive in some of their practices and that abuse needs to be recognized. They are not the Web Authorities and they don’t make the rules. When they do bad things to hurt the rest of us, they need to be chastized. But I also wanted to end the article on a positive note (for those of you who hang around to read the entire partially scannable thing).

Let’s explore these issues a little further. BTW — these criticisms and accolades should be considered by every search engine out there. Even if you’re just repackaging someone else’s search technology, you could do a lot to engage with your users (and you should avoid the abusive practices that might be open to you).

Imply that the United States Government approves of and supports your guidelines. (GOOGLE)
If any other search engine has tried to use the U.S. Government as an unwilling supporter for its policies, I am not aware of such a practice. I would object to any search engine doing this, not just Google. Even Danny Sullivan has criticized Googler Matt Cutts for invoking the U.S. Government in the war against paid links. The U.S. government’s position is that links are not endorsements and anyone who says otherwise is not speaking for the U.S. government.

Frankly, if the U.S. government ever regulates search engine guidelines we’ll all be in for a heap’o’trouble because the search engines will hire lobbyists and get in there before those of us who need to be heard ever have a chance to take action. That said, sometimes the search engine guidelines make some pretty outrageous claims (although they have generally made an effort to fix problems when the SEO community raises an issue with serious guideline problems).

For our part, the SEO community (including me) has been toxic on more than one occasion. Our exasperation can be exasperating and if we want the search engines to behave nicely we owe it to them (and their employees) to also behave nicely. Some days, I’m just not as nice as I should be. Sorry.

“Test” servers by making up fake URLs and crawling them. (YAHOO!, MICROSOFT)
Oooh! Sometimes you guys make me so mad I just want to stamp my feet! I understand why the search engines do this but they do it on such a massive scale it’s a form of spam. It’s intrusive, invasive, misleading, deceptive, and completely unjustifiable. If you want to crawl my servers, then just fetch the content that really exists. It’s not your place to ensure I set my HTTP header codes properly. It’s not your place to send me bogus traffic. It’s completely undefensible in the high court of moral opinion.

Artificially promote Web sites to the top of your search results. (ASK, GOOGLE)
Okay, I AM talking about Wikipedia — but I’m also talking about Universal Search in all its ugly forms. STOP THE MADNESS! This is not because some of my clients are irritated by the incredible nonsense you guys put up there. It’s because I’m sick of seeing obviously low-quality content shoved in front of better content.

More people use Google than Ask and there are lots of reasons for why Google sees so many more queries than other search engines. One of those reasons is the generally crappy quality in Google’s search results when I’m looking for specific detailed text and Google keeps throwing video, product search, image search, news search, blog search stuff in my face. I don’t want to see the pretty pictures when I’m looking for long, unique text strings.

I MIGHT want to see some suggested alternative searches (because, frankly, I probably don’t remember exactly what I’m looking for). You have no idea of how many times I’ve been asked to “look up such-and-such” because some jerk wants to flame me for making a sweeping generalization (which is all they do, but I digress). I can’t find stuff I was reading just the day before because “Query Deserves Freshness” and “The Index Is Rebuilding” and “This Is A Hot Topic” or “We Can Monetize The Heck Out Of This Query Here Are Five Ads In Your Face”.

You know who you are. You’re the search engines making my search experience awful. Stop that!

Focus on other search engines instead of your own. (MICROSOFT)
Steve Ballmer: SHUT UP ABOUT YAHOO! Just focus on building a great search engine with live.com. You’re doing fine.

Publicly humiliate people for not abiding by your guidelines. (GOOGLE)
Google takes a LOT of heat over all sorts of things. I get it. We’re constantly publicly humiliating Google. But here’s the thing: Google makes a big show of humiliating big brand sites for a month or two and then everything goes back to normal. I can do without those showcase outings because they may fool the masses but they don’t fool me.

Google is big into the whole “Hall of Shame” thing. They try to shame people into NOT issuing DMCA takedown notices by reposting those notices on ChillingEffects.Org. Google employs intimidation more often than it employs Fear, Uncertainty, and Dismay. It’s part of the Google process to intimidate people (how many public Toolbar spankings have we been subjected to now?).

These shameful shaming tactics don’t help the Web. They need to stop.

Report bogus data to Webmasters. (GOOGLE, YAHOO!)
Some of the bogus data is undoubtedly due to inevitable influences like crawl-to-reporting lag times, unforeseen bugs that creep into feature updates, etc. But there are certain bogus things the search engines report to Webmasters that just make no sense.

Yahoo!, why do you report links that don’t exist? Do those links count?

Google — why do your Webmaster Central links not match what is shown in your search index? You telling me that Webmaster Central does its own crawling? I can see a link reported as of January 1 for a page that doesn’t have a link in the cache because the cache was last updated on November 22. What’s up with that?

I think it’s important that if a search engine is going to report data to Webmasters then it should make both a good faith effort to syncronize the reported data with what Webmasters will find in the search results AND to explain the discrepancies. I have not noticed any problems with Microsoft but I’ll be honest — I don’t yet use the Microsoft Webmaster tools as much as I use Google and Yahoo!’s tools. That will change over time. If anyone knows of consistent discrepancies between Microsoft’s Webmaster reports and what they show in Web search, I would be interested to hear about it.

Change your service without explaining the change. (all four)
Geeze, they all do it. And given how fast search engines change over time, I understand how difficult it must be to get the marketing, corporate communications, and engineering departments to agree on any detailed message (God help those folks if they need executive buy-in as well).

Nonetheless, I’m not satisfied with “We’ve changed Blogsearch and we’re crawling more often, more deeply, blah blah blah”. What does that mean? Why does Blogsearch no longer report posts from active blogs (sometimes not for days, sometimes not at all)? Dudes, where are my favorite blogs?

But Ask, you keep dinking with your search results page format. I don’t care if you screw up crawlers — crawlers can be updated to match your updates. This is having an impact on my user experience. Why do you keep changing things? Tell me, tell me, tell me!

Here is what every major search engine could do to help the problem: add an interactive feature (that users can turn off) that explains what is going on when you change the way you serve search results. Just pop up your little mascot and have it say, “Hey! Notice the change? Click me to find out more!”

And then tell me why I’m no longer seeing the results I’ve come to expect. Why did you elect to remove certain types of sites? What can I do to get those sites back into my search experience (WITHOUT having to log in to personalized search)?

People use search engines for navigation and as notebooks. The search engines know this. You cannot go messing up people’s navigation and notebooks by “improving” the service. I use Google less for personal search now than I used to because, frankly, I can no longer find what I’m looking for in Google.

That’s a huge, major issue.

Refuse to publish (even supportive) comments on your blog. (ASK)
On more than one occasion I’ve posted comments to Ask’s blog in the past year. Sometimes I’ve been critical but there have been times when I was supportive. Earlier this week I posted a supportive comment. Where is my comment? Why are you not engaging? Why don’t you just turn off the comments feature?

Actually, I’ve noticed that Google sometimes refuses to let people comment. Shame on you, too, Google!

Automatically nofollow new user-generated content on your site. (GOOGLE)
I get the whole “we invested too much of our technology in link analysis and we’re scared people will abuse it” thing but really, Google, this makes you look silly. You roll out a new service and all the SEOs sign up to test it and we know right away that the links are nofollowed. You know, it’s really not all about links to the majority of your users. You’re not doing them any favors by automatically depriving them of editorial choice.

Share site-level information with Webmasters. (GOOGLE, MICROSOFT, YAHOO!)
I wish the data were more complete, more timely, and more accurate but I’m glad to have it. Ask, why don’t you offer a similar feature?

Announce new features to your visitors. (All four)
Only Google really does this right, but all four major search engines announce new features. I think that every search engine should commit to publishing a blog and putting content on that blog every week. Every day is great, but every week is fine. You have lots of information you can share with your visitors. Yes, I’ll be using it to learn more about you but I’m a user, too.

Blog often, blog broadly. (GOOGLE)
Sort of sounds like an echo. Google has reached out to Webmasters in so many aggressive ways, it’s no wonder the SEO community remains loyal to Google. Ask, Microsoft, and Yahoo! need to emulate this practice or find another way to communicate often and in a two-way fashion.

Offer secondary search capabilities (deeper, configurable site search; custom search). (GOOGLE, YAHOO!)
I don’t know what Ask’s fiscal plans are or what their technological capabilities are. Their search service is built on finding quality sites, not indexing the Web. Nonetheless, if they could offer site search they should.

Microsoft absolutely needs to address site search and custom search. Yes, they already offer site search (I even use it) but frankly it needs improvement. And there is room for innovation in site search. All the major search engines are overlooking a great opportunity to provide service to the Web community. Can it be monetized? I don’t know, but there is huge marketing value in supporting robust site search.

Answer questions directly on your site. (GOOGLE)
As I’ve mentioned before, the Google support forums generally offer bad advice but whenever a Googler drops by to explain a situation or offer feedback, it’s always helpful. Google’s recent “upgrade” makes it more difficult to find helpful information (although you can certainly find a lot of irrelevant and toxic comments) but that’s a whole different issue.

The point is that Google is reaching out to people and offering some actual feedback. I just wish they skipped the whole “let the blind SEOs lead the blind SEOs” phase and got down to brass tacks. When I suggested Google set up its own Webmaster forum a couple years back, I really meant they should set up an “Ask the Search Engineers” forum.

Every major search engine should do this.

Automatically nofollow new user-generated content on your site. (GOOGLE)
There is user-generated content and there is user-generated content. I think Google is setting a good example for other Web sites by showing people that you can allow UGC that doesn’t pass search engine value.

I believe Google should allow value-passing links in some context, but they are leading by doing rather than leading by dictating and that is a good thing for the Web community.

Engage with Webmasters through multiple conference circuits. (All four?)
I think they all do this. There are so many conference circuits out there I cannot imagine how the same search engineers would get to them all. But the more the search engines reach out to people, the better.

Use your own resources to promote yourselves. (ASK, GOOGLE, MICROSOFT, YAHOO!)
I don’t know when Ask added the ability to reach other Ask search services to their AskCity interface but they really lagged in the self-promotional area for a long time. Good navigation between the various vertical search tools is absolutely good for the Web community. Search interfaces should become ubiquitous.

Yeah, I get the whole “we’re competitors in a tight market” thing but you can and should establish brand value through your tools, your results, and your custom features. Your search interfaces should not be your brand value points. Really, the search interface is too important as a community environment to be compartmentalized. All telephones basically work the same way — we punch the numbers and make a connection with other telephones.

Search needs to be THAT simple.

There is always room for growth and improvement. Some improvements are really not good, in my opinion, but you cannot please all the people all the time.

The message here for all the search engines should be:

1. Be more transparent and open.
2. Be more consistent in your communications with users.
3. Provide more information to your users.
4. Treat Webmasters like partners in the Searchable Web Ecosystem.

There is room for improvement across the board, although obviously some search engines do better in one or more areas than the others.