Making the list is not the same as being chosen

If your venue is absent from an answer, or present but described wrongly, at least three different things could have gone wrong:

Separate problems, separate fixes, and one answer cannot tell you which one you have. A reply is a single draw from one system, one question, one place, one language and one moment.

In this study, what went with entering an answer was documentation: having its own website, review volume, a listed price, mentions elsewhere on the web. Star rating showed no association at that first step. Among venues that had already entered an answer, rating was associated with appearing in first position, and review volume still mattered there too.

Read narrowly: entry and rank behaved like two different outcomes and need measuring as two different things. None of these associations shows that changing a field causes a recommendation.

The paper is a public preprint and has not been peer reviewed. Its confirmatory wave, the main preregistered collection, covered 2,208 runs over seven days and produced 12,439 valid venue mentions, using 96 English queries, each phrased as if from a particular type of visitor, put to ChatGPT, Claude, Gemini and Perplexity through search-grounded APIs rather than their consumer applications. The design, hypotheses and analysis plan were preregistered publicly before the confirmatory data were collected, which makes it easier to separate planned tests from explanations chosen after the results were seen.

The entry associations are reported as odds ratios from a fitted model: review volume 1.64, a website of its own 1.92, a listed price 1.54, third-party web mentions 1.44, and rating 0.89, which the author reads as no association at that step. Within answers, rating was associated with first position at 1.17. An odds ratio above 1 means higher odds inside that model. It is not a percentage-point gain, and it is not evidence of cause.

Repeatability was measured too, and the numbers are easy to misread. Asking the same system the same question again returned a substantially different set of venues: mean Jaccard similarity between repeated runs ran from 0.22 for Claude to 0.45 for Gemini. Jaccard similarity is the share of venues two lists have in common out of every venue either list names, so 1 means the two lists are identical and 0 means they share nothing. A mean of 0.30 therefore says the typical pair of runs overlapped by roughly a third of the names involved. It does not say the same answer came back 30% of the time. Measured the same way, overlap among the top 20 venues across different systems ran from 0.33 to 0.54. A retest two weeks later produced turnover comparable to reruns inside the original period, which the author attributes to sampling variation rather than change over time.

The instrument had limits of its own. Repetition counts differed by system, with Perplexity run 10 times per query, ChatGPT and Gemini 5 times, and Claude 3 times. Claude's route was capped at two searches per run. After fixes, 91.5% of runs passed the strict validity check, and matching the names in answers to real venues was roughly 98% to 99% accurate rather than perfect.

Norly funded and conducted the study, and Norly sells review-management and AI-visibility tools to hospitality businesses. The author discloses that competing interest and describes the preregistration and validation steps intended to insulate the work. Those steps do not make the commercial conflict disappear. Obenan works in the same category, so this evidence, including our reading of it, deserves the same care.