SERP is an acronym for Search Engine Results Page. When you search for something, the resulting list of links shown is known as a SERP, and there is a lot of jockeying done to get placed high on those pages.

Search Today

As all of us here are, more than likely, long time denizens of the Internet. We have seen search evolve from extremely primitive to the complex, interactive, data-driven, service that it is today. Always there on some of our phones, and rarely more than a click, swipe, or gesture away. For those of us who remember the term, it is the killer app of the digital world.

It is November 27, 2020, at 6:25 PT as I write this, and Google Search Statistics is telling me that today Google has made 5.8 Billion searches. According to Statista, they have an 88% market share. Just using these two numbers as our data, today there will be at least 6.6 Billion searches. Perhaps other than breathing and certain other bodily functions, it might be one of the most frequent human activities.

In fact, in the digital domain, searching is as vital as air. Without search, the Internet is a giant, motile, scramble, rich with information that’s impossible for humans to decode or even find.

Just as polluted air eventually causes disease, polluted search results cause a variety of problems, some of which are only now becoming clear. Who knows what we’ll find out next year? Next decade?

At the moment, the air we’re breathing might be coming right out of a tailpipe. Are we all just so used to the fumes that it seems normal to us? What if there’s a better way? What if we could get clean air? Even as I write this, it seems almost anathema to even think about Search being different than it is today.

How could it be better

Stepping back from that precipice just a bit, let’s imagine for a moment what unpolluted search might look like.

I suggest it would have the following attributes:

• Always available. It would never be more than one action away — a click or a tap or a swipe.
• Unbiased. Independent. Results on Big Search are clearly polluted by business interests — why can’t I just get a query answered? If there are really only 4 results that closely match my query, why show me 70,000 or 5?
• Boolean. It should always be possible to search for p and q, p or q, or p and q not z.
• Deep. Results on the 100th SERP page do no one any good at all — what if that’s the answer I need?
• Transparent. Why is XYZ company at the top of my organic results? Today, no one can answer that question for any given search.
• Wildcard enabled. The term .jpeg should always give me all of the results that have .jpeg as their extension — especially true in search on my local system and the clouds I regularly access for storage.

And, more than almost anything else, it should be checkable. What if Big Search presents the wrong answer? How would you know? How would you check and see that their answer is right? There are simply not a lot of ways to do that today.

I’ve had the experience of getting incorrect results on all of the current search engines, as I am sure we all have — Why did they show me that? There is just no good way to get a view of the results of my query that’s different than Google, but just as good.

This goes to transparency and unbiased attributes. For instance, on the SERP below, there are no ads, so the only results are the results of Google’s ranking of the results of my query, but there is not one single person in the world who can tell me why Whatismybrowser.com is at the top of that list.

And there is no way to check. Even going to the site, I still don’t have any idea if they are at all reputable or have any financial agreement with Google. Why can’t we know why that site is there? Is something being hidden?

Source: Author/What browser am I using — Searched August 20, 2020

As with many things on the Internet, as search grew organically, it took on a lot of attributes that no one would find acceptable until, today, those features begin to dominate the whole experience. If we were starting to build search today, would we build a massive, complex system that reaches into almost every part of our lives?

Has Big Search become overly intrusive

Are they going beyond what is needed for good results?

Take a look at this book. It’s from the highly respected search engine Journal website called How Search Works. Look at page 59 and see just one sequence in the middle of a much longer sequence of events in providing your SERP. Admittedly, some of this is needed to provide the results we want and, these days, it’s time to wonder just how much of this is needed for us to get those results.

In the time it takes to provide a SERP, they have looked deeply into things like how you’ve interacted with websites before, and they use that, in some mysterious way, to determine what results are on your SERP. All in the name of personalized search, which is supposed to make your time searching more productive.

Does it? Or is the whole idea nonsensical? I can see why it’s better for Big Search, but just because I was looking for Macintosh computers yesterday doesn’t mean that I’m not searching for Macintosh apples today. The reasoning behind personalized search just happens to handily align with Big Search’s business interests — not our interests necessarily, and certainly not all the time.

Yes, the Internet is big and complex. But, why does it take How Search Works 148 pages to explain search? Shouldn’t search be a bit more straightforward? OK, sure, there are pictures and diagrams in the book, but still — Let’s think for a minute about what it might take to provide high-quality SERPs. For this essay, let’s say that a high-quality SERP is a list of links that allow me to answer my question quickly and accurately.

What’s the minimum amount a search engine could do to provide that for me? Isn’t that, in these days of anti-trust suits, what most of us are looking for?

There should be a Hippocratic Oath for search — cause no harm. Be minimally invasive.

Search in Action

Coming down just a bit from our 30,000-foot view, let’s see how some of the larger Search providers stack up against our six attributes of what’s needed for search — available, unbiased, boolean, deep, transparent, and wildcards.

Apple

Now, in the name of full transparency, I am an admitted Apple guy — very happy in my walled garden, thank you.

But search has been the turd in Apple’s punchbowl for a long time, and it’s not a lot better today. Spotlight on the Mac is a huge fail. And it gets worse if you try to include anything in the cloud.

On iOS general search is better but not in the native apps. To search in Message, or a number of other native apps, I have to scroll all the way to the top of the list. So if I am 300 or 3,000 messages down, that’s a lot of scrolling. It fails the always available test.

Google

The 6,000,000 pound gorilla. Consuming 98% of mobile search and something around 67% of fixed search, depending on who’s numbers you believe.

But is it good? What do you think?

It’s actually a difficult question to answer, isn’t it? It’s kind of like, for those of us old enough to remember this, asking if you liked TV when there was only The Big Three (ABC/CBS/NBC), consuming 98% of the airwaves. Until cable TV came along, there just wasn’t a lot of comparisons you could make.

The question isn’t really, is Google a good search engine? The question is, what is Google? Is it a search engine? Eric Schmidt said no several years ago — I’m still searching for the reference, but I read the original article. According to him, it’s an advertising platform that happens to serve search as its attraction. Same as The Big Three TV channels, they were advertising engines that served entertainment as their attraction.

Back to the question — is Google a good search engine? Well, a tech company, or any company, can only optimize for one thing at a time. If you’re optimized for serving ads, you’re not optimizing for search. It serves up searches pretty well, but I think all of us would agree that it’s a great advertising platform. But a search engine? We’ll see.

Even Brin and Page, in their senior thesis, said that advertising would pollute search.

Let’s put a pin in that one for the moment.

Amazon

Here is an even more dramatic version of you can only optimize for one thing at a time. Amazon should be a search engine, they would likely make even more money, but they keep showing me rows of women’s pajama bottoms when I search for men’s. They’re wasting my time and their resources.

Bing

Bing is the index that almost all the non-Google search engines use. Duck-Duck-Go (DDG), for instance, uses Bing, as do the nascent search engines, such as Neeva and Qwant.

The problem is, well, just search the same thing on Google and compare the results to any of the Bing-run search engines. The results from the Bing index are just unsatisfying.

From a developer standpoint they are a lot easier to work with than Google, but in every instance, DDG or any of the Bing-driven search engines, the results are a subset of what Google can show you.

The problem is that the Bing index is limited. None of the search engines publish real numbers about how many pages they have indexed, so it’s a bit of a mystery, but the available data seem to point to Google’s index being the mother of them all.

These all seem to be part of the answer to why search sucks. Some of the implementations are funky and break the all-important always available attribute. But the unbiased question still looms large. There is no doubt that Big Search’s SERPs are influenced by their business interests. We just have no idea how.

Photo by Maxim Ilyahov on Unsplash

Other attributes

Deep is an interesting topic. I believe that Big Search is trying hard to provide good results in the depths of its SERPs, such as any page higher than 4.

Certainly, the revenue incentive is reduced that far down for the businesses — they’re making less money down there, so there is less reason to skew the results. Big Search is good for broad, shallow searches, but they don’t do depth too well. Searching deeply into a specific topic frequently requires a different approach, as researchers will tell you.

How could we fix that? What about a set of easily accessible and understandable controls that could turn the butter knife into a laser scalpel? Big Search has little to no incentive to offer this. But it sure would be valuable to those of us who are more than just shallow searchers.

Ultimately it comes down to a question of how to surface content that appears low-value to most people but is really important to me? In order to provide that, a search engine would have to have a different business model, one not driven by advertising — back to Paige and Brin’s early work.

Wildcard enabled. The major search engines do a good job of this. Where this is lacking is when you off the engines and go to sites that don’t search well, that things break down. Most of us have had the frustration of being on a company’s site, forced to search down their funnel rather than just getting what I want. A trick I have learned is to go back to Big Search and use the site: operator, for example, search for site: nyt.com swisher and trump, and you get results just from the NYT on anything that has both names.

So I hope that someday soon, every CEO of every company has to find something on his own company’s support page as a consumer. Maybe it should be a national holiday.

Here, for instance, is an example of the support page of one of my favorite apps — references deleted because it is one of my favorites:

Image courtesy of the author

What’s missing? A way to find what I want without paging through every article until I find what I need. Search.

Let’s address the unbiased attribute for a moment because I think it really important. It’s at the core of the distrust around search. Here’s what Sergey Brin and Larry Page had to say in 1998:

The goals of the advertising business model do not always correspond to providing quality search to users.

That may just be truer today than ever before and harder to determine.

How much are Google’s financial incentives polluting the results they provide us? We may never know. How the algorithms work is secret. What biases the results are hidden and completely out of our control. And now Google couldn’t change its business model even if it wanted to. But that’s what it would take to provide search that causes no harm and is minimally invasive.

Perhaps, now that this is becoming so apparent, we can start asking for something better.

If Yahoo was Search 1.0, Google is Search 2.0.

Perhaps it’s time for Search 3.0?