Been thinking about this a lot [1]. Will this fundamentally change how people find and access information? How do you create an experience so compelling that it replaces the current paradigm?
The future promised in Star Trek and even Apple's Knowledge Navigator [2] from 1987 still feels distant. In those visions, users simply asked questions and received reliable answers - nobody had to fact-check the answers ever.
Combining two broken systems - compromised search engines and unreliable LLMs - seems unlikely to yield that vision. Legacy, ad-based search, has devolved into a wasteland of misaligned incentives, conflict of interest and prolifirated the web full of content farms optimized for ads and algos instead of humans.
Path forward requires solving the core challenge: actually surfacing the content people want to see, not what intermiediaries want them to see - which means a different business model in seach, where there are no intermediaries. I do not see a way around this. Advancing models without advancing search is like having a michelin star chef work with spoiled ingredients.
I am cautiously optimistic we will eventually get there, but boy, we will need a fundamentally different setup in terms of incentives involved in information consumption, both in tech and society.
I agree that this is the core question, but I'd put it as: Who Gets To Decide What Is True?
With a search paradigm this wasn't an issue as much, because the answers were presented as "here's a bunch of websites that appear to deal with the question you asked". It was then up to the reader to decide which of those sites they wanted to visit, and therefore which viewpoints they got to see.
With an LLM answering the question, this is critical.
To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously). But there are many places around the web saying other things (which is why my friend was confused). An LLM trawling the web could very conceivably come up with a non-truthful answer.
This is possibly a bad example, because the truth is very clearly written down by the government, based on exact laws. It just happened to be a recent example that I encountered of how the internet leads people astray.
A better example might be "is dietary saturated fat a major factor for heart disease in Western countries?". The current government publications (which answer "yes") for this are probably wrong based on recent research. The government cannot be relied upon as a source of truth for this.
And, generally, allowing the government to decide what is true is probably a path we (as a civilisation) do not want to take. We're seeing how that pans out in Australia and it's not good.
> To paraphrase a recent conversation I had with a friend: "in the USA, can illegal immigrants vote?" has a single truthful answer ("no" obviously)
Er, no, the meaning of the question is ambiguous, so I'm not sure "has a single truthful answer" is accurate. What does "can" mean? If you mean "permitted", then no. But if you mean can they vote anyway and get away with it? It's clearly happened before (as rare as it might have been), so technically the answer to that would be be yes.
This is a fundamental limitation of language. The LLM is likely to provide a good answer here even though the question is technically ambiguous, because it likes verbose answers.
Equally "can" is used to substitute for other precise words. Humans are good at inferring context, and if someone asked me "can illegals vote" I'd say "no". Just like if someone said "can you pass the salt" I pass the salt, I don't say "yes".
If the inferred context US wrong then the "truth" is wrong, but as with talking to humans it's possible to refine context with a follow up question.
I'm no linguist, but the question does seem unambiguous, or quite clear, to a reasonable observer. The context is "voting in a US election" AND the subject is "an illegal immigrant" WITH an assumption that the illegal immigrant has, in fact, illegally emigrated to the US.
At the risk of derailing the conversation down a completely different rabbit hole... As I understand it, only citizens are legally entitled to vote, and voting requires a government-issued ID and the voter to be enrolled.
How did they vote and get away with it previously?
(also, as per another comment, if you know that this happened then surely they didn't get away with it?)
If they got away with it then how do you know it's happened?
Neal Stephenson's book _Fall, or; Dodge in Hell_, from 2019, dedicates many words to this concept. Briefly summarized, it explores a post-truth-world, describing the world where people could agree on the truth as a narrow time-slice in history. People have their own individual internet filters, and the USA becomes divided into Afghanistan-etc-like tribes, each an echo chamber. (Ameristan)
In the book, âThe Big Changeâ (1952), Frederick Allen talks about the year 1900, and (among many differences) he notes that for nearly everyone in the year 1900, the limits of their world rarely extended beyond their own town.
We have this idea today that everyone online is getting trapped in echo chambers, but thatâs been the case for most of human history.
Hah, I'm not the only one that brings up this book in this context.
The way this is wrought, in the novel, is a savant engineer writes a bot framework that can cheaply and quickly disseminate torrents of misinformation about a provided subject, and then open sources this framework. He basically broke the internet on purpose as a sort of accelerationist move I suppose.
> A better example might be "is dietary saturated fat a major factor for heart disease in Western countries?". The current government publications (which answer "yes") for this are probably wrong based on recent research. The government cannot be relied upon as a source of truth for this.
I know it was just an example, but actually no, the role of dietary saturated fat as a factor for heart disease remains very much valid. Iâm not sure which recent studies youâre referring to, but you can't undo over 50 years of research on the subject so easily. What study were you thinking about?
>you can't undo over 50 years of research on the subject so easily
Sure you can if the research was bogus to begin with, sponsored in many cases, and merely taking for granted/referencing some previous results without verifying them, which is often the case.
You can undo 50 years of research easily, by this process which we call science.
I think they are referring to low-carb studies done recently. If your diet consists of only saturated fat, it does seem to be healthier for you than the standard American/Western diet that is also high in saturated fat but also quite high in sugar, wheat, and other starchy carbs. When combined, saturated fat and carbs are a hitting a double if your goal is to be unhealthy.
General disclaimers apply regarding portion sizes etc blah blah blah I'm not a doctor.
Anecdotally, a low-(ish) carb diet and fasting has done wonders for my health and many others. I will say that there appears to be a link with higher cholesterol when consuming higher amounts of fat, but the argument in nutrition science atm seems to be centered on whether or not that is "good" cholesterol, but it's hard to measure in human patients for a long time because you essentially need to put them on a very limited diet to get good data. Those large scale trials are expensive and hard to manage at scale.
https://pmc.ncbi.nlm.nih.gov/articles/PMC9794145/
Remarkable how easy it is to cling to propaganda
What is truth anyway? I see it as a quicker version of browsing to web to get a summary of what people say. As you said, with search you get a bunch of websites where strangers talk about a certain topic. You read a dozen and see if they agree with each other or if they sound legit.. There is just a huge overlap between what we consider true and what (an overwhelming majority of) people agree on. A lot of things are reduced to the consensus. If you ask a non-obvious question, you usually get an answer "a lot of people you consider trustworthy dedicated some time studying this question and agreed that the answer is X". But then people can be wrong, a big group of people can be wrong, people can be bribed or perhaps you don't actually trust these people that much. The internet can't tell you the truth. LLM can't tell you the truth. But they can summarize what other people in the world say on this subject.
I think to solve this problem we need to take a step back and see how a human would solve the problem (if he had all the necessary information).
One thing is clear, in the vast majority of cases we don't have a single truth, but answers of different levels of trustworthiness: the law, the government, books, Wikipedia, websites, and so on.
A human would proceed differently depending on the context and the source of the information. For example, legal questions are best answered by the law or government agencies, then by reputable law firms. Opinions of even highly respected legal professionals are clearly less reliable than government law itself and are likely to be a point of contention/litigation.
Questions about other facts, such as the diameter of the earth or the population of a country, are best answered by recent data from books, official statistics and Wikipedia. And so on and so forth.
If we are not sure what the correct answer is, a human would give more information about the context, the sources, and the doubts. There are obviously questions that cannot be answered immediately! (if it's to easy to find the truth, we would not need a legal system or even science!) So no machine and no amount of computation can reliably answer all questions. Web search does not answer a question. It's just trying to surface relevant websites to a bunch of keywords. The answering part is left as an exercise for the user ;)
So an AI search with a pretense of understanding human languages makes the task incredibly harder. To really give a human-quality answer, the AI not only needs to understand the context, but it should also be able to reason, have common sense, and be a bit self-aware (I'm not sure...). All this is beyond the capabilities of the current generation of AI. Therefore, my conclusion is that the "search" or better said the "answer" cannot be solved by LLM, no matter how much they fine-tune and tweak it.
But we humans are adaptable. We will find our way around and accept things as they are. Until next time.
I genuinely think Kagi has led the way on this one. Simplicity is beautiful and effective, and Kagi has (IMHO) absolutely nailed it with their AI approach. It's one of those things that in hindsight seems obvious, which is a pretty good measure of how good an idea is IMHO.
Google could have done it and kind of tried, although they're AI sucks too much. I'm very surprised that OpenAI hasn't done this sooner as well. They're initial implementation of web search was sad. I don't mean to be super critical as I think generally OpenAI is very, very good at what they do, but they're initial browse the web was a giant hack that I would expect from an intern who isn't being given good guidance by their mentors.
Once mainstream engines start getting on par with Kagi, there's gonna be a massive wave of destruction and opportunity. I'm guessing there will be a lot of new pay walls popping up, and lots of access deals with the search engines. This will even further raise the barrier of entry for new search entrants, and will further fragment information access between the haves and have-nots.
I'm also cautiously optimistic though. We'll get there, but it's gonna be a bit shakey for a minute or two.
> I'm also cautiously optimistic though. We'll get there, but it's gonna be a bit shakey for a minute or two.
But I don't understand how all of these AI results (note I haven't used Kagi so I don't know if it's different) don't fundamentally and irretrievably break the economics of the web. The "old deal" if you will is that many publishers would put stuff out on the web for free, but then with the hope that they could monetize it (somehow, even just with something like AdSense ads) on the backend. This "deal" was already getting a lot worse over the past years as Google had done more and more to keep people from ever needing to click through in the first place. Sure, these AI results have citation results, but the click-through rates are probably abysmal.
Why would anyone ever publish stuff on the web for free unless it was just a hobby? There are a lot of high quality sites that need some return (quality creators need to eat) to be feasible, and those have to start going away. I mean, personally, for recipes I always start with ChatGPT now (I get just the recipe instead of "the history of the domestication of the tomato" that Google essentially forced on recipe sites for SEO competitive reasons), but why would any site now ever want to publish (or create) new high quality recipes?
Can someone please explain how the open web, at least the part of the web the requires some sort of viable funding model for creators, can survive this?
> Why would anyone ever publish stuff on the web for free unless it was just a hobby
That's exactly what the old deal was, and it's what made the old web so good. If every paid or ad-funded site died tomorrow, the web would be pretty much healed.
The internet was great before the great monetization of it, had tons of information provided for free with no ads. After ads, it will still have tons of information. Stack Overflows will still exist, Wikipedias, corporate blogs that serve just to boost the company, people making courses and other educational content, personal blogs (countless of which make their way here), all of those will continue to exist.
Ad-driven social networks will continue to exist as well.
The age of the ad-driven blog website is probably at an end. But there will be countless people posting stuff online for free anyway.
Many people have an intrinsic motivation to share knowledge. Have a look at Wikipedia. There are enough of these people that we don't need to destroy the open Internet to accommodate those who only write when they expect to be paid.
> the history of the domestication of the tomato" that Google essentially forced on recipe sites for SEO competitive reasons
That may help with SEO, but another reason is copyright law.
Recipes can't be copyrighted, but stories can. Here is how ChatGPT explained it to me:
> Recipes themselves, particularly the list of ingredients and steps, generally can't be copyrighted because they're considered functional instructions. However, the unique way a recipe is presentedâsuch as personal stories, anecdotes, or detailed explanationsâcan be copyrighted. By adding this extra content, bloggers and recipe creators can make their work distinctive and protectable under copyright law, which also encourages people to stay on their page longer (a bonus for ad revenue).
> In many cases, though, bloggers also do this to build a connection with readers, share cooking tips, or explain why a recipe is special to them. So while copyright plays a role, storytelling has other motivations, too.
Yep, I was incredibly skeptical about Kagi but I tried it and never looked back. Now my wife, friends, and several coworkers are customers.
The chatgpt approach to search just feels forced and not as intuitive.
Once Kagi implements location aware search that is actually useful Iâll be interested in Kagi. Thatâs what made me leave the engine besides loving it otherwise.
I wouldn't usually point this out, but as you did it repeatedly: "they're" is a contraction of "they are". You're looking for the possessive, "their".
- Your local grammar pedant
I gave Kagi a shot two weeks ago, and it instantly impressed me. I didn't realize how much search could be improved. It's a beautiful, helpful experience.
Yeah, itâs wonderful. Especially once you take the time to up/downrank domains.
> Will this fundamentally change how people find and access information? How do you create an experience so compelling that it replaces the current paradigm?
I think it's already compelling enough to replace the current paradigm. Search is pretty much dead to me. I have to end every search with "reddit" to get remotely useful results.
The concern I have with LLMs replacing search is that once it starts being monetized with ads or propaganda, it's going to be very dangerous. The context of results are scrubbed.
> The concern I have with LLMs replacing search is that once it starts being monetized with ads or propaganda, it's going to be very dangerous.
Not to mention that users consuming most content through a middle-man completely breaks most publishers business models. Traditional search is a mutually beneficial arrangement, but LLM search is parasitic.
Expect to see a lot more technical countermeasures and/or lawsuits against LLM search engines which regurgitate so much material that they effectively replace the need to visit the original publisher.
> Traditional search is a mutually beneficial arrangement, but LLM search is parasitic.
Traditional search is mutually beneficial... to search providers and publishers. At expense of the users. LLM search is becoming popular because it lets users, for however short time this will last, escape the fruits of the "mutually beneficial arrangement".
If anything, that arrangement of publishers and providers became an actual parasite on society at large these days. Publishers, in particular, will keep whining about being cut off; I have zero sympathy - people reach for LLMs precisely because publishers have been publishing trash and poison, entirely intentionally, optimizing for the parasitic business model, and it got so bad that the major use of LLMs is wading through that sea of bullshit, so that we don't have to.
The ad-driven business model of publishing has been a disaster for a society, and deserves to be burned down completely.
(Unfortunately, LLMs will work only for a short while, they're very much vulnerable to capture by advertisers - which means also by those publishers who now theatrically whine.)
Yes, I also don't understand how LLM based compaines expect people to keep producing contect for them for free.
âFuck you, pay meâ - Childish Gambino
The whole thing needs a reframe. Ad driven business only works because its a race to the bottom. Now we are approaching the bottom, and its not gonna be as competitive. Throwback to the 90s when you paid for a search engine?
If you can charge the user (the customer- NOT the product) and then pay bespoke data providers (of which publishers fall under) then the model makes more sense, and LLM providers are normal middlemen, not parasites.
The shift is already underway imo - my age cohort (28 y/o) does not consume traditional publications directly. Its all through summarization like podcast interviews, youtube essays, social media (reddit) etc
Same here?
Search means either:
* Stackoverlow. Damaged through new owner but the idea lives.
* Reddit. Google tries to fuck it up with âAuto translationâ?
* Gitlab or GitHub if something needs a bugfix.
The rest of the internet is either an entire ****show or pure gold pressed latinum but hardly navigatable thanks to monopolies like Google and Microsoft.PS: ChatGPT already declines in answer because is source is Stackoverflow? AndâŚwellâŚthese source are humans.
I've become so complacent these last 20 years. I wonder if I try to browse the web, will I stumble upon anything as awesome as the 1998 web scene was?
> Search is pretty much dead to me.
I've heard reports that requesting verbatim results via the tbs=li:1 parameter has helped some people postpone entirely giving up on Google.
Personally I've already been on Kagi for a while and am not planning on ever needing to go back.
Fuzzy search is cancer. I search for $FOO, click a result, Ctrl-F for $FOO ==> Not found. Many such cases. If there's a way to force DuckDuckGo to actually do what I tell it to, I'd love to hear it.
> I think it's already compelling enough to replace the current paradigm. Search is pretty much dead to me. I have to end every search with "reddit" to get remotely useful results.
I worry that there's a confusion here--and in these debates in general--between:
1. Has the user given enough information that what they want could be found
2. Is the rest of the system set up to actually contain and deliver what they wanted
While Aunt Tillie might still have problems with #1, the reason things seem to be Going To Shit is more on #2, which is why even "power users" are complaining.
It doesn't matter how convenient #1 becomes for Aunt Tillie, it won't solve the deeper problems of slop and spam and site reputation.
This is probably Google's Altavista moment, by making their results crappier by the year in search of Ad dollars everyone has felt that there is a potential for search to be better and once that becomes available they'll be in a continious game of catch-up.
Yes, Google has their own AI divisions, tons of money and SEO is to blame for part of their crappiness. But they've also _explicitly_ focused on ad-dollars over algorithmic purity if one is to believe the reports of their internal politics and if those are true they have probably lost a ton of people who they'd need right now to turn the ship around quickly.
At some point it seems like Google switch to ML-based search instead of index based search. You can search for very specific combinations of lyrics and scenes: "eyes on me pineapple bucket of water house of cards chess time loop" and you won't surface a link to the music video featuring all of those things (https://www.youtube.com/watch?v=AlzgDVLtU6g), you'll just get really generic results of the average of your query.
Has google completely stopped working for anyone else?
I can still search things, i get results but, they're an ordered list of popular places the engine is directing me to. Some kind of filtering is occurring on nearly every search i make that's making the results feel entirely useless.
Image search stopped working sometime ago and now it just runs an AI filter on whatever image you search for, tells you there's a man in the picture and gives up.
Youtube recommendations is always hundreds of videos i've watched already, with maybe 1-2 recommendations to new channels when i know there's millions of content creators out there struggling who it will never introduce me to. What happened to the rabbit holes of crazy youtube stuff you could go down?
This product is a shell of its old self, why did it stop working?
For me it feels like Google is adding "buy" keyword to every search.
Part of the answer is here - https://www.wheresyoured.at/the-men-who-killed-google/
I see this argument in HN a lot, so I checked my search history (googled "search history") it seems I use it ~10-20 times a day, looked at individual searches e.g. last week, and except a few queries, I have found what I was looking at.
Yes it is hard to find some stuff in internet because it is filled with generated affiliate spam and walled gardens, Has Google stopped working for me? Nope It seems still alive and kicking.
Google search is completely broken IMO. I stopped using Google search years ago and every time I go back on the off chance that it's bigger index has something that DuckDuckGo couldn't find for me.
Image search isn't great either but it still often gives me something close and that usually satisfies my image searching needs.
I still find YouTube recommendations quite good for me, but there are occasional ones I've watched already. I still go down its fun (and educational!) rabbit holes all the time.
Today I tried finding a rather popular product I used recently, which I had forgotten the name. I had to search THREE TIMES before finding it:
First search (âproducts that do Xâ) got me a bunch of those comparisons sites, none of them containing the one I was trying to find
Second search (âycombinator startup that does Xâ) got me a page of spam, but at least I found the product name
Third search (company name) got me an ENTIRE PAGE of ads and SEO optimized pages before the actual link to the actual product
FWIW, ChatGPT Search didn't surface the video either with that query
> Based on the elements youâve describedâeyes, a pineapple, a bucket of water, a house of cards, chess, and a time loopâitâs challenging to identify a single music video that encompasses all these features.
Searching for a song by typing in lyrics is specifically a feature Iâve noticed Google canât do any more. And it used to be able to.
Same here. I've generally found complaints about Google's recent decline to be overblown, but lyrics are one area where it's truly gone off the deep end.
Even if I type in 2-3 lines worth of nearly-exact lyrics that show up on multiple lyrics websites, it'll give me completely unrelated songs that match several words at most.
On that note, if anyone needs a good browser-based song finder, the "Aha Music Identifier" extension is pretty good. It's a life saver when watching Twitch streams that don't list the currently-playing song.
Not just lyrics. Searching for quotations or text excerpts is much harder and double quotes to only get exact matches also returns unrelated results.
On the other hand, chatgpt 4o-mini and 3.5 will just make up source material which is amusing but not very helpful.
It works only if they're well-known lyrics or a popular track
> Google switch to ML-based search
The very earliest form of page rank used a form of ML.
> This is probably Google's Altavista moment
Yes and no. Yes for the quality of search results. Google algorithm and user experience was simply better than AltaVista's, but Google had another advantage. It used a cluster of cheap consumer-grade hardware (PCs) as its backend, rather than the expensive servers AltaVista used, in fact, AltaVista started off as a way for DEC to show off its hardware.
As a result Google was not only better than its competitors at searching, it was also cheaper and scaled better, and this architecture became the new standard.
It is the opposite for these AI-based systems. AI is expensive, it uses a lot of energy, and the hardware is definitely not consumer-grade.
What it means is that barring a significant breakthrough, cheap ChatGPT-like services are simply unsustainable. Google don't have to do anything, it will collapse on its own. Ironically, probably in the same way that Google result became crappier by the year, but on fast forward.
> What it means is that barring a significant breakthrough, cheap ChatGPT-like services are simply unsustainable.
This is the basic premise of Ed Zitron's article https://www.wheresyoured.at/to-serve-altman/ and others written around the same time. A lot of what he's written seems to be coming to pass, e.g. unfathomably large investments (~$100 billion) by tech giants in OpenAI and other AI startups to keep them going. That does seem unsustainable.
Can anyone make a counterpoint to this claim? I'd be very interested to hear what a viable path to profitability looks like for OpenAI.
Probably "every major corporation pays them a gazillion dollars a year for enterprise-class AI services that outperform humans while running at 100x speed"?
Google's SEO almost killed the web, ChatGPT Search will finish the job.
How?
How can an economy function if there is no way to get any value from producing anything?
Google (and Facebook, a few other platforms) made it so that the vast majority of websites are never visited. ChatGPT further erodes the possibility of tying economic value to the production of that value.
Seems like we need a new framework around "intellectual property"
In essence there is no âobjectiveâ algorithm. Itâs a mish mash of random black box patterns that we hope delivers the right answer. It probably will most of the time but it will be gamed hard and itâs going to be hard to undo changes in gamified models.
We are still spending most of our time online on social media sites like Hacker News, Instagram and Youtube, right?
That's not the web
For me the key difference between ChatGPT Search and Google is the feedback mechanism.
With ChatGPT, I can give a thumbs up or thumbs down; this means that OpenAI will optimize for users thumbs up.
With Google, the feedback is if I click on an Ad; this means that Google optimizes for clickbait.
Youâre also giving Google feedback when you click a search result link. Presumably that should be a huge signal for measuring search quality.
Heck, Google even promoted the `ping`[0] anchor attribute feature so they can log what link you click without slowing you down. (Firefox doesnât support ping, which means when Firefox users click on a Google search result link theyâre sent to an internal google.com URL first and then redirected for logging purposes)
[0] https://developer.mozilla.org/en-US/docs/Web/API/HTMLAnchorE...
I have no doubt that Googleâs search team is optimizing for the best results. The problem is their ads team is optimizing for revenue. You canât optimize for two things at the same time without compromising (the optimum is the Pareto frontier).
In my opinion, the issue is that the user's definition of "best results" and Google's definition do not align, including the Search team. Google's incentives are very different than user's needs.
Is it optimising for all users? And assuming people thumbs up the correct info. I wonder what accuracy percentage we are looking at there. ChatGPTs responses are so confident when wrong I fear people will just give it a thumbs up when its wrong. (This is if how I understand the feature you mention is working)
I am not talking about accuracy. Only experts can determine factuality.
I am talking about relevance or returning what I asked. If I ask for reviews for SaaS product, Google will usually return a rival vendorsâ biased review.
If ChatGpt search returns a review written by âthe professional association of xxx developersâ or another unbiased site, I will give it a thumbs up. I believe other people will do the same.
No way that gets gamed
If the current iteration of search engines are producing garbage results (due to an influx of garbage + SEO gaming their ranking systems) and LLMs are producing inaccurate results without any clear method proposed to correct them, why would combining the two systems not also produce garbage?
The problem I see with search is that the input is deeply hostile to what the consumers of search want. If the LLM's are particularly tuned to try and filter out that hostility, maybe I can see this going somewhere, but I suspect that just starts another arms race that the garbage producers are likely to win.
Search engines tend to produce neutral garbage, not harmful garbage (i.e. small tidbits of data between an ocean of SEO fluff, rather than completely incorrect facts). LLMs tend to be inaccurate because in an absence of knowledge given by the user, it will sometimes make up knowledge. It's plausible to imagine that they will cover each other's weaknesses: the search engine produces an ocean of mostly-useless data, and the LLM can find the small amount of useful data and interpret that into an answer to your question.
The problem I see with this "cover for each other" theory is that as it stands having a good search engine is a prerequisite to having good outputs from RAG. If your search engine doesn't turn up something useful in the top 10 (which most search engines currently don't for many types of queries) then your llm will just be summarizing the garbage that was turned up.
Currently I do find that Perplexity works substantially better then Google for finding what I need, but it remains to be seen if they're able to stay useful as a larger and larger portion of online content just AI generated garbage.
> Search engines tend to produce neutral garbage, not harmful garbage (i.e. small tidbits of data between an ocean of SEO fluff, rather than completely incorrect facts)
Wasn't google AI surfacing results about making pizza with glue and eating rocks? how is that not harmful garbage?
That's not a plausible imagination that such a prefect complement exists
You just described the value proposition of RAG.
Garbage-ness of search results is not binary, the right question is: can LLMs improve the quality of search results? But sure, it won't end the cat and mouse game.
I think that's the right broad question. Though LLMs properties mean that for some number of cases they will either make the results worse, or more confidently present wrong answers. This prompts the question: what do we mean by "quality" of results? Since the way current LLM interfaces tend to present results is quite different from traditional search.
> it won't end the cat and mouse game.
There is no way to SEO the entire corpus of human knowledge. ChatGPT is very good for gleaning facts that are hard to surface in today's garbage search engines.
The question is what is the business model and who pays for it, that determines how much advertising youâre getting. It is not clear if OpenAI could compete in Ad-supported search. So maybe OpenAI is trying to do the basic research, outcompete the Bing research group at Microsoft and then serve as an engine for Bing. Alternatively they could be just improving the ability of LLMs to do search, targeting future uses in agentic applications.
If I can pretty quickly tell a site is SEO spam, so should the LLM, no? Of course that would just start a new round in the SEO arms race, but could work for a while.
> If I can pretty quickly tell a site is SEO spam, so should the LLM, no?
Why would you assume that?
The LLM is not a human and cannot distinguish between spam and high quality content.
It seems pretty trivial to fine tune a model to do this - this is really playing to LLM's strengths.
Try perplexity and then come back and tell us how you feel
Every time I see one of these topics, I go ask chat GPT a question to which I know the answer on a topic where I would like to be able to get useful answers to similar questions that I do not know the answer to.
This time it was, "Did Paul Edwin Zimmer write a fourth Dark Border novel?" (Real answer: Yes, Ingulf the Mad. You can find the answer on his Wikipedia page.[1])
ChatGPT's[2] answer: "Yes, Paul Edwin Zimmer wrote a fourth novel in the Dark Border series titled "The Dark Border." This book was published after the original trilogy, which included "The Dark Border," "The Gilded Age," and "The Silver Sphere." If you're interested in the themes or plot, let me know!" (Note: these are not the titles of the 2nd and 3rd novels in the series. Also, it gave me the same name for the putative 1st and 4th books.)
Pure hallucination.
1. https://en.wikipedia.org/wiki/Paul_Edwin_Zimmer 2. https://chatgpt.com/
ChatGPT 4o and 4o-mini with Search were able to answer this question without any issues. Maybe you didn't enable the search functionality?
---------------
4o:
Yes, Paul Edwin Zimmer wrote a fourth novel in his Dark Border series titled Ingulf the Mad, published in 1989. This installment focuses on the characters Ingulf Mac Fingold and Carrol Mac Lir, detailing their meeting and acquisition of mystical swords. Notably, Istvan Divega, the protagonist of the earlier books, does not appear in this novel.
---------------
4o-mini:
Yes, Paul Edwin Zimmer authored a fourth novel in his Dark Border series titled Ingulf the Mad. Published in 1989, this book shifts focus from the previous protagonist, Istvan DiVega, to explore the backstory of Ingulf Mac Fingold and Carrol Mac Lir, detailing their initial meeting and the acquisition of their mystical swords.
The complete Dark Border series consists of four novels:
1. The Lost Prince (1982)
2. King Chondos' Ride (1983)
3. A Gathering of Heroes (1987)
4. Ingulf the Mad (1989)
These works delve into a world where the Hasturs' power is crucial in containing dark creatures, and the narrative unfolds through various characters' perspectives.
Why is it that comments complaining about flaky AI responses never share the chat link, and then invariably someone replies with examples of the AI answering correctly but also fail to share the chat link?
A lot of these responses are so funny. "Um, when _I_ asked this question to ChatGPT it gave the right answer!" Yeah, because it's flaky! It doesn't give the same answer every time! That's the worst possible outcome!
Funny, yes. But... a webforum is a good place for these back-and-forths.
I'm on the other side of this fence to you. I agree that the conclusion here is that "it is flaky." Disagree about what that means.
As LLMs progress, 10% accuracy becomes 50% accuracy. That becomes 80% accuracy and from there to usable accuracy... whatever that is per case. Not every "better than random" seed grows into high accuracy features, but many do. It's never clear where accuracy ceilings are, and high reliability applications may be distant but... sufficient accuracy is not necessarily very high for many applications.
Meanwhile, the "accurate-for-me" fix is usually to use the appropriate model, prompt or such. Well... these are exactly the kind of optimizations that can be implemented in a UI like "LLM search."
I'm expecting "LLMs eat search." They don't have to "solve truth." They just have to be better and faster than search, with fewer ads.
Isn't it even a bit interesting that GP has tried it every time something new has come out but not once gotten the expected answer? Not only that but gets the wrong titles even though Search for everyone else is using the exact Wikipedia link given in the comment as the source?
LLMs are run with variable output, sure, but it's particularly odd if GP used the search product as it doesn't have to provide the facts from the model itself in that case. If GP had posted the link to the actual chat rather than provided a link to chatgpt.com (???) I'd be interested in seeing if Search was even used as that'd at least explain where such variance in output came from. Instead we're all talking about what could have happened or not.
Normally I would agree with you but for such a fuzzy topic involving search it would require all pages and discussions it can find to be the same and not outdated. I don't see why anyone would presume these systems to be omnipotent.
Several folks have already mentioned that ChatGPT with search returns the correct answer to your question (with a source to explore directly).
I really think this latest release is a game changer for ChatGPT since it seems much more likely to return genuine information than ChatGPT answering using its model alone. Of course it still hallucinates sometimes (I asked about searching tabs in Firefox Mobile and it told him the wrong place to find that ability while citing a bunch of Mozilla help docs), but it's much easier to verify that by clicking through to sources directly.
It feels like a very different experience using ChatGPT with search turned on and the "Citations" right side bar left open. I get answers from ChatGPT while also seeing a bunch of possibly relevant links populate. If I detect something's off I can click on a source and read the details directly. It's a huge improvement on relying on the model alone.
My gut reaction here is that the hallucination is caused by how you [rightfully] formed the prompt. GPT has no way of reliably determining what the fourth book is, so it infers the answer based on the data provided from Wikipedia. I'll bet if you changed the prompt to "list all books by Paul Edwin Zimmer", it would be incredibly accurate and produce consistent results every time.
Its true.
I usually seed conversations with several fact-finding prompts before asking the real question I am after. It populates the chat history with the context and pre-established facts to build the real question from a much more refined position.
Genuine question: is there a present or planned value proposition for people like me who already have decent search skills? Or are these really for children/elders who (without making any normative claim about whether this is a good thing or not) can't be arsed to perform searches themselves?
Does someone else have good search skills but mingle traditional search engines with LLMs anyways? Why?
I use LLMs every day but wouldn't trust one to perform searches for me yet. I feel like you have to type more for a result that's slower and wordier, and that might stop early when it amasses what it thinks are answers from low effort SEO farms.
I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find. Traditional search engines (I'll jump between Kagi, DuckDuckGo, and Google) haven't proved as useful at pointing me in the right direction when I find that I need to spend a few sentences describing what I'm looking for.
LLMs on the other hand (free ChatGPT is the only one I've used for this, not sure which models) give me an opportunity to describe in detail what I'm looking for, and I can provide extra context if the LLM doesn't immediately give me an answer. Given LLM's propensity for hallucinations, I don't take its answers as solid truth, but I'll use the keywords, terms, and phrases in what it gives me to leverage traditional search engines to find a more authoritative source of information.
---
Separately, I'll also use LLMs to search for what I suspect is obscure-enough knowledge that it would prove difficult to wade through more popular sites in traditional search engine results pages.
> I find myself being unable to search for more complex subjects when I don't know the keywords, specialized terminology, or even the title of a work, yet I have a broad understanding of what I'd like to find.
For me this is typically a multi-step process. The results of a first search give me more ideas of terms to search for, and after some iteration I usually find the right terms. Itâs a bit of an art to search for content that maybe isnât your end goal, but will help you search for what you actually seek.
LLMs can be useful for that first step, but I always revert to Google for the final search.
Also, Google Verbatim search is essential.
Yeah this is exactly how I use LLMs + Google as well. I would even go further and say that most of the value of Google to me is the ability to find a specific type of source by searching for exact terminology. I think AI search is fatally flawed for this reason. For some things generic factual information is okay ("What's the capital of France?") but for everything else, the information is inextricably bound up with it's context. A spammy SEO blog and a specialist forum might have identical claims, but when received from the latter source it's more valuable, it's just higher signal.
Google used to care about this but no longer does, pagerank sucks and is ruined by SEO, but it still "works" because if you're good you can guess the kind of source you're looking for and what keywords might surface it. LLMs help with that part but you still need to read it yourself, because they don't have theory of mind yet to make good value judgements on source quality and communicate about it.
I also find some use for this. Or I often ask if there's a specific term for a thing that I only know generally, which usually yields better search results, especially for obscure science and technology things. The newer GPTs are also decent at math, but I still use Wolfram Alpha for most of that stuff just because I don't have to double check it for hallucinations.
You might like what we're building in that sense :D (full disclosure, I'm the founder of Beloga). We're building a new way for search with programmable knowledge. You're essentially able to call on search from Google, Perplexity other search engines by specifying them as @ mentions together with your detailed query.
I don't overuse LLMs for now; however when I have a complex problem that would require multiple of searches and dozens of tabs opened and reading through very long docs, asking LLM allows me to iterate order of magnitude faster.
Things that were previously "log a jira and think about it when I have a full uninterrupted day" now can be approached with half an hour spare. This is game changer because "have a full day uninterrupted" almost never happens.
It's like having a very senior coworker who knows a lot of stuff and booking a 30m meeting to brainstorm with them and quickly reject useless paths vs dig more into promising ones, vs. sitting all day researching on your own.
The ideas simply flow much faster with this approach.
I use it to get a high level familiarity with what's likely possible vs what's not, and then confirm with normal search.
I use LLMs also for non-work things like getting high level understanding of taxation, inheritance etc laws in a country I moved in, to get some starting point for further research.
This. Not having to open two dozen tabs and read through so much is a gamechanger, especially for someone who has had trouble focusing with so much open. This is especially true when learning a new technology.
I dunno, I'm not exactly on the AI bandwagon, but search is the one place where I use (and see others using) chatgpt all the time. The fact that Google search has been getting worse for a decade probably helps, but better search -- consistently done, without ads or cruft -- would be worth a few bucks every month for me.
I agree that you can't TRUST them, but half the links regular search turns up are also garbage, so that's not really worse, per se.
Same, but, until recently, I've been using Microsoft's Co-Pilot because for the longest time it did exactly what this new "search" feature added to ChatGPT: it produced a list of source material and links to reference the LLM's output against. It was often instrumental for me and I did begin to use it as a search engine considering how polluted a lot of first-search results have become with spam and empty, generated content.
Oddly, Microsoft recently changed the search version of Copilot to remove all the links to source material. Now it's like talking to an annoying growth-stage-startup middle manager in every way, including the inability to back up their assertions and a propensity to use phrases like "anyway, let's try to keep things moving".
Happy to see this feature set added into ChatGPT â particularly when I'm looking for academic research in/on a subject I'm not familiar with.
I find that my search skills matter less and less because search engines try to be smarter than me. Increasingly I am confronted with largely unrelated results (taking tweaked keywords or synonyms to my query as input apparently) as opposed to no results. So my conclusion is that the search engines increasingly see the need of search skills as an anti pattern they actively want to get rid of.
On the Google search results page, activate Search tools > All results > Verbatim. You can also create your own search provider bookmark with verbatim search as the default by adding âtbs:li=1â as a query parameter to the Google search URL.
Completely agreed. At a certain point, âskillsâ became fighting a losing battle with Google incessantly pushing me towards whatever KPIs or ads theyâre chasing. Itâs a poor use of my effort and time to keep chasing what Google used to be.
What a lot of people miss is that it's not replacing Google. It's creating a different way of interacting with the web. For example, I spent last night brainstorming a random business idea that involved international logistics. Because of this realtime Search feature, at some point in that conversation (so it didn't start as a 'let's search some web stuff', just organically happened) The topic of warehousing came up. So I asked it to find me suitable warehouses in 2 different countries and compare pricing. In a heartbeat, I was looking at something very specific, tangible and available. So a 'classical ai chat' became tangible because of this feature. I've had a similar experience a few more times with this feature. So 'searching the web' wasn't even a concept in my head, I was just deep in a conversation and ChatGPT basically went 'Let me find you X' and it just presented it. Google, or any classical search based engine, requires me to have that conversation in my head - then get to a point where my intent is clearly articulated - then I go and insert any search term.
OpenAI also provides a Chrome Extension (https://chromewebstore.google.com/detail/chatgpt-search/ejcf...) to trigger a search from the omnibar as the default search engine.
If you don't like that (like I do), you can also manually add it under Site Search using
First impression: Far too slow to replace Google Search for this use. I frequently get 5+ seconds of waiting before the first word of a response shows up, vs. less than 1 for Google (which is not as fast as it used to be). OpenAI has a lot of work to do on latency.
I can definitely see this new search feature being useful though. The old one was already useful because (if you asked) you could have it visit each result and pull some data out for you and integrate it all together, faster than you could do the same manually.
It's often hobbled by robots.txt forbidding it to visit pages, though. What I really want is for it to use my browser to visit the pages instead of doing it server side, so it can use my logged-in accounts and ignore robots.txt.
I canât help but wonder if the lag time in chatgpt responses is artificial and intentional, to prevent people spamming requests.
I tried hacking this together a month ago as an experiment and it was super painful. This seems like exactly what I wanted - props to OpenAI. Google should be on DEFCON 2.
How did you find the extension? Searching the Chrome Web Store for 'SearchGPT' turns up a ton of third-party extensions, not made by OpenAI. Also, there doesn't appear to be a way to search by developer.
They presented the link in the ChatGPT interface. Maybe this was an A/B-test thing, idk.
sama agrees search for extensions not easy lol: https://x.com/sama/status/1852163907735162937
Makes me question why Google never bothered to create something like search sessions which could be enriched with comments/notes and would be located in a sidebar just like the chats in ChatGPT/Claude/Mistral are.
They really had the potential to do something interesting, but were just focused on their ad metrics with the "good enough" search box. What have they been doing all the time?
the FAANG giants have been government assets for ~15+ years [0]. they don't have to turn a profit every quarter, innovate, or make their search any better because they no longer play by the rules a normal business does. they are a critical "too big to fail" component of the state's global surveillance system.
[0] https://static1.makeuseofimages.com/wordpress/wp-content/upl...
Linking the slide deck that caused Google to start encrypting the traffic between their own data centers running on their own fiber is perhaps not the most compelling argument that Google is a state asset.
https://www.newyorker.com/news/amy-davidson/tech-companies-s...
You think it is so hard for the NSA to have a tap within Google's datacenters?
You think Google could say no to NSA if they were asking nicely to put a tap?
Encryption between datacenters is to keep away other state actors, not the US.
> they don't have to turn a profit every quarter,
And, yet, aside from Aramco, they are the most profitable companies in the history of the world.
> they don't have to turn a profit every quarter,
What does this mean? Like, I work there and Iâd be pretty annoyed if they stopped turning a profit as a collapse of the stock price would affect my compensation.
Itâs interesting to hear this take because Iâm used to hearing the opposite: that Google is too focused on increasing short-term profit at the expense of product quality.
OpenAI is Microsoft. Microsoft is a FAANG giant.
Microsoft is their leading investor. They don't technically own OpenAI.
Microsoft is literally not in "FAANG". But they are in MANGA.
How is that relevant? Microsoft bought OpenAI, didn't create it by R&D, so the assertion stands: giants don't do new things, for whatever reason.
I guess now Google's search stack is too complicated and not many engineers understand what to do in order to introduce a novel, big feature integrated into the full stack vertically and horizontally. And those few capable of doing so are probably completely out of bandwidth, so some random ambitious PM cannot pull their hands into uncertain green field projects.
Collecting people's data and making money from that.
Google doesn't make money from "collecting people's data", they show you ads.
If they're collecting data it doesn't even work; I make no effort to hide from them and none of their ads are targeted to me. Meta, though, they're good at it.
How do you think Google's ad business works?
Chrome did add a sidebar that shows search sessions (queries grouped with the pages you visited on that topic). Used to be called "Journeys". I don't think you can add notes. I never found it useful in the slightest and I doubt notes would have made it any better. Chrome has been adding random UI features like that over time, but I haven't found any of them at all useful in many years.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.