Search is broken.

Picture it: Me, in my office, yesterday afternoon. As a freelance writer, I get paid to research specific subjects so I can provide content that my clients think is useful and for which they pay me the dough that funds my vacation in Las Vegas this week. Much of what I write about for one particular client focuses on the tech industry — trends, products, user tips, how-to narratives, etc.

No matter whether I use Google or Bing (I prefer the latter), the outcome of Web searches is usually the same: I get bupkis for the first dozen or so pages of results unless the search term is exceptionally narrow. Or rather, I get what the Wise and Mighty Algorithm thinks I ought to get — products, or SEO-optimized content mill filler. Or Wikipedia.  Oh, and increasingly, I get to see what equally ill-informed people in my social networks think about the search terms … yay. Nothing like the blind leading the clueless.

Yesterday provided a case study in what’s wrong with search. My task was to pull together roughly 1,000 words (augmented with authoritative online third-party sources, as needed) about spy gear that works with laptops. Try searching for “spy gear” or even “laptop surveillance” or any permutation of keywords you wish. What do you find? Yup: Product placement pages.

Oh, there’s plenty of stuff online if you craft a sufficiently narrow search. But if you already know the right mix of search terms that gets you exactly what you’re looking for right out of the gate, then you probably don’t need to engage in search in the first place. In my case, I had no desire whatsoever to buy equipment; I wanted to see comparative uses of equipment with an eye toward the legal ramifications of third-party surveillance. (Yes, I tried at least two dozen different searches to zero in on what I wanted. It shouldn’t be that much of a chore.)

Experts lament the filter bubble and wax indignantly about the role of SEO optimization, but the problem really relates not to end-user behavior but by the practices of the search engines themselves. The data-hungry search servers want to know everything about you — not to give you better results, but to better sell ads that target you. The relevance of search results feels more and more like an afterthought, and the moral preening of folks like Matt Cutts at Google (who’s in the business of setting the algorithm that picks winners and losers on the Internet — i.e., “Panda”) reinforces the point that at heart, search isn’t about knowledge discovery, it’s about market segmentation.

Two major problems degrade the usefulness of modern search results. First, the drive toward personalization leads the algorithm to send you information you’ve already decided you liked — but this problem can be fixed by logging out of Google or Bing and deleting search histories or browsing using a private mode. Really, my searches yesterday shouldn’t influence the results of today’s search for wholly unrelated content. It’s like going to the local library, heading to the card catalog, and the librarian greets you at the door with a stack of cards for Stephanie Meyer and Ann Rice books because yesterday you looked up a history text about Vlad the Impaler. Oh, and while you’re here, would you like to buy a Twilight T-shirt?

Second, rankings are structured to reward site developers that have countered your Google-fu with various SEO hacks. This problem is harder to escape, and it’s made worse by the presence of engine-approved advertising. The argument that product-heavy results dominate the the rankings because people search to buy products feels like question begging, especially when you recall that the search engines make their money based on ad impressions.

In a perfect world, a search engine would empower the user to discover exactly what he’s looking for. To some degree, such a wish still fails in the absence of a robust semantic Web; search needs to understand the difference between content and metacontent much more robustly than it currently does.

But consider how useful an intuitive roll-your-own algorithm would be.  Instead of chucking phrases into a box and hitting “search,” trusting in the accuracy and integrity of an unknown algorithm that doesn’t have your best interests at heart, you could specify exactly what you want. You would add key terms but then provide aggregation context — e.g., “I’m looking for academic information” versus “I’m shopping” versus “I am looking for news analysis or commentary.” Because you’re still going to get many, many results, you could also specify the ranking logic: “Give me the most recent stuff first” or “give me content from authors with a Klout score of greater than 30″ or whatever. The end user gets to control how data is sliced, diced and presented.

Right now, Google and Bing provide “advanced search” capability. In practice, though, these tools are only marginally useful. Searchers can stipulate the language they’re looking for or limit searches to specific top-level domains, but you can’t tell the engine to emphasize or suppress product sites or content-mill drivel.

In a perfect world, search engines would help you discover what you’re looking for. In the real world, search engines give you what you think you deserve.

Like I said: Search is broken.

• «Previous Post
• Next Post »

---