The Curious Case of Keyword Search (And What We’re Doing About It)

Quick: what's changed in technology in the last twenty-five years?

The (obvious) answer is: pretty much everything. From cell phones, to email, to that ear-splitting dial-up internet tone… it's more difficult to name what hasn'tchanged.

However, there is one thing that has thrived over the last 20-plus years without changing much at all: Keyword Search. Now, you're probably thinking that search has changed dramatically since the '90s. And, you're right. Consumer search has seen dramatic improvements in that period of time, thanks to Google. But there is one area of search that is stuck in a bit of a time warp, yet has never been more popular: Keyword Search. Keyword Search powers the majority of business websites, but it is lightyears from the experience you get when searching on Google.

The question is: why? Why has Keyword Search hung around for so long, and haven't businesses buried it in the tech graveyard along with fax machines and landlines? To answer those questions, we first need to go back in time to 1994 — and the invention of the (second) most important Keyword Search engine ever built.

Hyperlinks? As if!!

At the dawn of the "world wide web," the year 1994 marked the explosion of Keyword Search. You know, the ability to type a keyword into a search bar — say, "puka shell necklace" (if you really want to go full '90s) — and actually findlinks to pages that mentioned puka shell necklaces.

At the time, this was a huge step forward. In a mere 36 months, the internet saw dozens of Keyword Search engine launches, including Infoseek, Yahoo!, Lycos, Webcrawler, Looksmart, Excite, and AltaVista. Then, in 1998, Google launched PageRank, which turned out to be the best algorithm to rank web results for keyword searches on the consumer internet.

But this is where most search stopped improving — a full 23 years ago.

That's because around the same time, former Xerox PARC engineer Doug Cutting launched an open source Keyword Search engine called Apache Lucene.

Why was this such a landmark moment? Well, with Lucene, enterprise developers could add a keyword search engine to their websites, enterprise applications, help desks, ecommerce sites, enterprise apps and more. Basically, if you wanted to let people search on your own website — not just on Google or Ask Jeeves (RIP) — you could finally do it.

Even today, odds are that you use Lucene-powered search almost as often as you use Google-powered search; you just don't know it. Lucene (or ElasticSearch, which is built on Lucene) powers the search that happens on most of the websites you visit.

But here's the problem: while there wasn't much of a difference in the '90s, Google search in 2021 is vastly superior to Lucene-powered Keyword Search.

This is because Google pioneered a breakthrough in search technology called Natural Language Processing or NLP.

The missing key from Keyword Search

NLP is a branch of artificial intelligence (AI) related to how software can digest, understand, and manipulate language. Put simply, it's all about the ability of a computer program to understand human language in the way that it is spoken.

NLP understands questions like "When was Marco Polo born" and tells you "September 15, 1253." This answer comes from the Google Knowledge Graph.

Keyword Search, on the other hand, would be looking for the keywords you typed — like, literally, the word "Polo." And those kind of search experiences almost always give you a set of blue link results — based on individual keywords — that rarely help you find what you want. It's happened to all of us, right? You see the magnifying glass on a website, type in a keyword, and get a set of terrible, irrelevant results back, so you give up and try something else to find what you want.

To be clear: we all still type in or say keywords when we search. We might ask about Marco Polo by saying "Marco Polo born." Not exactly a full question — and that's ok, because NLP understands the intent. The outdated part is the keyword search platform, which is unable to detect the difference between a series of keywords and a series of keywords that are implicitly in question form.

What's the alternative? A truly modern, Google-like on-site search experience. Modern search has these four elements:

1. Uses NLP powered by advanced AI to understand natural language questions, like the one above
2. Uses multiple algorithms to present different sets of results
3. Is in a dynamic user interface (UI)
4. Is built on a Knowledge Graph

Let's quickly break down those four elements.

The Input

Keyword Search works like "Ctrl-F" in a Word Document. But the problem is that a lot of times you don't search with keywords that exactly match what you need.

So, in a Keyword Search situation, a search for "Physicians in Vienna, VA" gives you no results back.

That's because there's no place on their entire website that has the exact string of words "Physicians in vienna, va" — and that's a pretty bad experience.

Instead, modern search uses AI-powered Natural Language search to fundamentally understand the user's input — giving you the ability to understand very precise questions like, "Physicians in Vienna, VA who accept Aetna and speak spanish."

Single vs. Multi algorithm

The "Control F" Keyword Search approach works fine for a reasonably sized document like a research paper or even a 500 page book, as the user can just chronologically "Find Next" through the document.

But the world wide web has exploded to billions of pages. Check this out: the keyword "Marco Polo" for example, is mentioned in 207 million places — way too many for anyone to sort through. These places need to be ranked. So the Keyword Search engines of the late '90s adapted by focusing on their ranking *algorithm.*To sort through these 200+ million results, search engines needed a strong algorithm that ranked hyperlinks by relevance.

But modern search uses this (lengthy) list of blue links only as a last resort.

For example, the Google SERP for the keyword "McDonalds" displays 500 million results, but only a single web link at the top: McDonalds.com. The rest are barely visible at the bottom. Google combines snippets, maps, knowledge cards, and other elements to present the user with multiple options for their results.

Each of those different elements uses a different algorithm — and that's how modern on-site search should work. For each query that comes in, you need to apply multiple algorithms to the question, and give the user back a dynamic result set that best matches what they've asked for.

Hyperlinks vs Answers

Keyword Search literally provides a list of hyperlinks, ranked by relevance, meaning that you have to click the link and read the results. This doesn't complete your quest — it just points you in the right direction.

Modern search completes the search loop by answering user questions directly, and allowing them to transact as the next step.

Users get multiple elements to answer their questions. Sometimes these are snippets from extractive QA. Or sometimes it's a list from the knowledge graph, like maps or a list of people, and sometimes they are direct answers from the KG.

Best of all, when the user gets their results, they can transact right off the SERP. Depending on what you ask, your result set offers different kinds of transactions: you can order online, or request directions, or call, or buy a product.

In search of better experiences

Modern, multi-algorithm, answers-based search is better Keyword Search from the late '90s on every dimension. That's why we developed our Answers Platform.

With a full AI-powered, answers-led search platform, enterprises can now build a single, structured knowledge graph for their company, and deliver answers – not links – everywhere people search: in smart services like Google and Siri, and with natural language search in their own websites, support sites, and ecomm.

So say goodbye to Keyword Search of the past, and hello AI search built for today.

Click here to learn more — or schedule a demo.