Accuracy of AI search overviews

[AllThingsTech]

Was debating on whether or not to post this in Air India thread but felt it was insensitive to derail the topic given the sensitivity of the subject so posting here. Also makes for a good discussion on member experiences with AI overviews, and mishaps imo.

AI overviews frequently incorrectly blame Airbus rather than Boeing for this plane crash.

AI Overviews hallucinates that Airbus, not Boeing, involved in fatal Air India crash

Google’s disclaimer says AI “may include mistakes,” which is an understatement.

arstechnica.com

This is likely because many articles on this plane crash cite Airbus to be Boeing’s main competitor; the inability of AI to comprehend context is a real hazard. What’s interesting is that according to the article, a percentage of the time you’ll receive Airbus as the answer, and other times you’ll receive Boeing. It’s a real mixed bag - outputs aren’t fixed based on the same inputs.

What’s your experience like with AI overviews? Have you generally found it to be accurate? Identified anything nonsensical?

Also who should be held responsible for libel/slander when AI damages a person/organisation’s reputation? Should disclaimers protect the AI companies, or should they be held accountable for misinformation that inflicts damage to reputation on said entities?

 

[Retro]

Oh yeah, Google's AI Overviews does sillies like this all the time. For example, I've just watched Salem's Lot (2004) a reasonable horror movie by Stephen King. I looked up "Is Salem's Lot a real place" and it came up with lots of info about it. Among that, it stated, correctly, that it's fictional, but incorrectly that Salem's Lot used to be called Salem's Lot. Oh dear. I looked really carefully at the spelling to check I didn't miss some small detail.

This thread is in the correct place, regardless of the unfortunate connection with the Air India crash.

 

[AllThingsTech]

Among that, it stated, correctly, that it's fictional, but incorrectly that Salem's Lot used to be called Salem's Lot. Oh dear.

Looks like you made a typo Oh dear
Or was this nonsensical sentence the entire point?

 

[Retro]

No, the AI literally spelled it the same way twice.

 

[AllThingsTech]

No, the AI literally spelled it the same way twice.

 

[AllThingsTech]

 

[AllThingsTech]

@Retro please can you edit the thread title to "Accuracy of AI search overviews"
I missed the word "search"
Otherwise, it sounds like I'm giving an overview of AI hehehe

 

[Retro]

Done.

 

[live627]

The AI overview usually does not like to make sense of my complex and niche queries. Instead, it opts for a more basic response that is related.

 

[AllThingsTech]

It concerns me that being presented with an AI search overview reduces the likelihood of click on links, including the ones included in the references of the AI search overview, and increases the probability of the user ending their browsing session. This means they're unlikely to assess the validity, the precision of the information displayed.

https://www.pewresearch.org/short-reads/...e-results/

What's more, the most frequently cited sources in both Google AI summaries and standard search results are Wikipedia, YouTube and Reddit, not the most trustworthy sources of information. On the other hand, the vast majority of these summaries (88%) cited three or more sources. Only 1% cited a single source.