# Google vs. *ChatGPT*

An online experiment with 212 participants on how classic and AI-based information search are perceived. The short answer: it depends on how much is at stake.

> Project page: https://clemenshoeller.at/en/projects/google-vs-chatgpt
> Category: research/experiment

## Key facts

- **Type** · Bachelor's thesis
- **Institution** · WU Vienna
- **Date** · June 2025
- **Scope** · Sole author
- **Result** · 4 of 5 hypotheses confirmed

- Full thesis as PDF ↗: https://clemenshoeller.at/projects/bachelorarbeit-suchverhalten-google-chatgpt.pdf

## /01 The question

Information search sits at the start of almost every purchase decision. For two decades that meant Google: type a query, scan a list of results, pick a source. Dialogue-based systems like ChatGPT change that form fundamentally, because they return an answer instead of a selection.

The interesting question is therefore not which system is better, but what each is better for. Consumer research offers a useful lever here: **involvement**, meaning how much personal relevance and perceived risk a decision carries. Under high involvement people process information systematically and check sources; under low involvement what matters is that it is quick and easy.

That led to the guiding question: does the preference between Google and ChatGPT shift depending on how much is at stake? Five hypotheses were derived from the literature:

1. **H1** For high-involvement searches Google is preferred, because it is credited with greater trustworthiness, currency and traceability.
2. **H2** For low-involvement searches ChatGPT is preferred, because the search feels more efficient and more pleasant.
3. **H3** Regardless of involvement, ChatGPT is experienced as more helpful for the decision and is reused or recommended more often.
4. **H4** ChatGPT produces higher subjective decision confidence, because its recommendations are structured and tailored.
5. **H5** ChatGPT is perceived as more personalized regardless of involvement.

## /02 Method

A controlled online experiment in a **2×2 design**. The first factor was the search method (Google or ChatGPT), the second the involvement level of the task. Each person was randomly assigned exactly one of the four combinations and actually completed the task before rating it.

### The two tasks

- **Low involvement:** find a café suitable for working (Wi-Fi, power sockets, good coffee). Everyday, low risk, no deep research needed.
- **High involvement:** find a suitable general practitioner (reviews, opening hours, accessibility, personal care). High personal relevance, high need for information.

### Sample & analysis

Assignment ran automatically through SoSci Survey as a rotating procedure (drawing without replacement). 212 interviews were completed in full, unevenly distributed because of drop-outs after assignment: the two ChatGPT tasks were finished slightly less often.

- ChatGPT · Low: 49 participants
- Google · Low: 56 participants
- ChatGPT · High: 44 participants
- Google · High: 63 participants

Ratings used standardized seven-point scales. For each hypothesis, three to four matching items were aggregated into a mean and compared with an **independent t-test**. In addition, a **two-way ANOVA** per hypothesis checked whether the difference really came from the platform rather than from individual items or their interaction. All analyses were run in GraphPad Prism 10.

## /03 Results

Four of the five hypotheses were supported. The pattern is consistent: Google wins on trust, ChatGPT on experience and personalization, and on the question of which system people would use again, nothing separates them.

*ChatGPT · Google*

- Trust & information quality *H1 · High-Involvement*: 4.26 · p < 0.0001 η² 0.18
- Efficiency & experience *H2 · Low-Involvement*: 6.19 · p < 0.0001 η² 0.099
- Reuse & recommendation *H3*: 5.33 · p = 0.488 not significant
- Decision confidence *H4*: 5.14 · p = 0.0083 weak effect
- Perceived personalization *H5*: 6.14 · p < 0.0001 η² 0.263

*Aggregated means per hypothesis on a seven-point scale (1 = do not agree at all, 7 = fully agree). The axis starts at 1 because that is the floor of the scale. p-values from the independent t-test, η² as effect size.*

### What sits inside the numbers

**H1, supported.** For the doctor search Google leads ChatGPT by 1.37 points, with a strong effect size (η² = 0.18). The ANOVA attributes 18.3% of total variance to the platform choice alone, and all four individual items (trust, absence of doubt, currency, traceability of the source) differ significantly. The result therefore does not rest on a single criterion.

**H2, supported.** For the café search the picture flips: ChatGPT leads by 0.90 points, a medium effect size. Three of the four items are significant; only "easy to use" is not, which makes sense, since nobody finds a Google search too complicated.

**H3, not supported.** 5.33 versus 5.41, p = 0.488, effect size essentially zero. Despite the clear experiential advantages, participants would not reuse or recommend ChatGPT more often than Google. This is the most interesting result of the thesis, because it shows how much habit weighs: a better experience in a single task does not yet shift the default choice.

**H4, supported but weak.** A 0.30-point difference in favor of ChatGPT, significant in the overall test (p = 0.0083), yet not one individual item reaches significance on its own. The effect is real but only emerges from the aggregate.

**H5, clearly supported.** The strongest finding of the thesis: a 2.07-point lead for ChatGPT, η² = 0.263, and all six individual comparisons (three items across both involvement levels) are highly significant. Personalization is the one point where the dialogue model looks consistently superior to the list of results.

## /04 Interpretation

How digital search systems are perceived depends heavily on the decision context. There is no better system, there is a better fit.

For digital marketing practice one implication stands out: content should no longer be optimized for search engines alone, but also for dialogue-based systems, because a growing share of research happens there and different rules apply. At the same time H3 shows that trust and habit move more slowly than the user experience. Anyone concluding that classic search will disappear in the short term is reading past the result.

> **Limitations** — The sample was not randomly recruited but gathered through social media, and consists predominantly of digitally affine people. Only two scenarios were studied, café and doctor search, which limits transferability to other products. Each person also performed only a single search interaction, whereas real research is usually multi-step. And with ChatGPT the quality of the answer depends heavily on how the prompt is phrased, so differences in participants' linguistic expression may have influenced the ratings.

### What I took away

The craft: setting up an experiment properly, deriving hypotheses from literature rather than from intuition, aggregating scales, applying t-tests and ANOVA correctly, and reporting results even when a hypothesis does not hold. Failing to confirm H3 was uncomfortable while writing and turned out to be the most valuable part of the outcome.

**Skills gained:** Experimental design · Data analysis & statistics · t-tests & ANOVA · Literature reviews · Academic writing · GraphPad Prism

## /05 Related projects

These projects connect more tightly than the tiles on the home page suggest. Here are all the others, sorted by how closely they relate to this one:

- [LLMs & Creativity](https://clemenshoeller.at/projekte/llms-kreativitaet)
- [Prompt Injection](https://clemenshoeller.at/projekte/prompt-injection)
- [Healthcare Fraud Detection](https://clemenshoeller.at/projekte/fraud-detection)
- [AI Automations](https://clemenshoeller.at/projekte/ki-automatisierungen)
- [AI Talent Gap](https://clemenshoeller.at/projekte/ai-talent-gap)
- [Consulting Cases](https://clemenshoeller.at/projekte/consulting-cases)
- [AURORA](https://clemenshoeller.at/projekte/aurora)
- [Zotter Marketing Plan](https://clemenshoeller.at/projekte/zotter-marketingplan)
- [This Website](https://clemenshoeller.at/projekte/diese-website)
- [VWA: Origin of the Internet](https://clemenshoeller.at/projekte/vwa)

---
*German version: https://clemenshoeller.at/projekte/google-vs-chatgpt.md*
*Bachelor's thesis (WU Vienna, 2025): an online experiment with 212 participants comparing classic and AI-based information search. Method, results and interpretation.*