1/ To orient ourselves, let’s use the classic IR taxonomy of web searches from Broder:

2/ Navigational: There’s one trusted site that has the information you need. Just get me to it as quickly as possible. If that site isn’t the #1 result, you are a bad search engine.

3/ Navigational: I don’t want an algorithmically generated synthesis, I just want to go to the site I have been to before or know has the answer I need. Search engine textboxes and the browser URL window merged for a reason.

4/ Navigational: ChatGPT’s response to most of these types of queries are “I’m sorry, but I am not able to browse the internet or access current information…”

5/ Navigational: You can get ChatGPT to return a navigational website if you ask in a particular way - “What is the website for the Clemson Tigers?” - and you get more info around that,

5.1/ but Google is highly optimized for this type of query today (both for definitive and hubs) so ChatGPT doesn’t really help here.

6/ Navigational: Search engines end up hard coding a bunch of these definitive type answers to make up for holes in the algos. That’s another example of 20 years of refinement that LLMs don’t address.

7/ Transactional: ChatGPT has no facility to help you mediate a transaction with a 3rd party. If I want to buy a specific model of tennis shoe, I need an ecommerce site that sells them, not a treatise on what tennis shoes are or the attributes to look for in a good one.

8/ Transactional: If I’m shopping or looking to access specific databases of information on the web, LLMs don’t help. Google has to ingest or contract with many entities to gain access to quality data for these queries.

9/ Transactional: A search for “PRS Silver Sky SE” on Google shows the effort that goes into incorporating shopping into search results. 3rd party dbs, tailored UI, reviews, price fluctuations, etc. ChatGPT responses can be an interesting part of this whole, but can’t replace it.

10/ Informational: these types of queries are really where ChatGPT can improve the speed and relevance of the experience. The system can synthesize the right result without me hunting through 10 different relevant sites and formulating the response myself.

10.1/ Of course, this all depends on the accuracy of the result and with it still being in its infancy, we see we can be fooled by it bullshitting us with answers that sound superficially good but are clearly inaccurate to a domain expert.

11/ Informational: ChatGPT punts on any query that requires real time data. Flight tracking, weather, etc. experiences aren’t improved with it. And again, a ton of work goes into creating tailored experiences for these based on trusted 3rd party databases.

12/ Informational: I do fear for the lack of human ingenuity we are baking into these systems. Having recently read Stuart Russell’s excellent book (https://www.amazon.com/Human-C... reducing search down to the one, definitive answer given by an algorithm with limited objective functions

12.1/ can be a recipe for dumbing us all down. Search, synthesis, and leveraging true language and domain understanding to formulate the best answer. Maybe this emerges from statistical methods like LLMs and maybe it doesn’t.

13/ Also consider what search engines call “full page relevance” vs. just focusing on “the 10 blue links”. There is a lot more on the page than the 10 algorithmic links. Maps, News, Images, Video, instant answers like weather, Books, Finance, Shopping, etc.

14/ We’ve all been trained to use the least amount of high information words in today’s search engines.I bet the average query in Google today is < 6 words.

14.1/ One thing LLMs might help with is to evolve this to a better conversational approach where the interaction feels more naturally human vs adapting to the system.

15/ When I was a PM on the ranking and relevance team for what became Bing, we always talked about how great it would be to ask “one more clarifying question” of the searcher. When you searched for “Tigers”, did you mean the animal, or a sports team from Clemson or Detroit?

16/ 💡 Evolving ChatGPT to be truly conversational where the system not only answers queries but uses its own knowledge of where its model is weak and past search session relevance to get the searcher to clarify one thing and dramatically improve relevance and speed to result.

16.1/ Turn the LLM from passive to active participant in the conversation.

17/ Big companies are slow, but when the threat affects >90% of your singular, dominant revenue, you pay attention. I’m sure the smart folks at Google already have query classifiers that are determining what % of queries can be better served or augmented by LLM approaches

17.1/ and are figuring out a way to introduce them inline to better serve the searcher and evolve what is already a highly optimized, and very profitable, results page.

18/ It is truly exciting to see the rapid improvements in LLMs and ChatGPT. They are a portent of amazing things to come.

19/ It’s more likely they get incorporated into existing holistic search experiences than kill the dominant search engine as they can only answer a % of the queries people make by the billions every day.