The chat was noticed, and welcomed
Nine of twelve users spotted the concierge immediately, and responses met expectations for ten of twelve. No one rated the experience poor or very poor.
IBM · Gen AI · Unmoderated study
Would anyone actually talk to the AI concierge?
Chat became a core entry point for product discovery and support on the Watson data integration and observability page. Before investing further, we needed to know whether people noticed the concierge, understood it, and trusted it enough to use.
12
participants across 7 countries
9/12
noticed the chat immediately
87%
rated the experience good or excellent
3/12
found the AI privacy statement
What we asked
Discoverability first, capability second.
- Can users notice and discover the chat concierge when they land on the product page?
- Do they recognize its purpose quickly enough to decide when to engage?
- Are the interactions — hover, click, prompts — simple and intuitive?
- Where does confusion show up repeatedly?
How we ran it
- Method
- Unmoderated prototype testing via UserTesting
- Participants
- 12 users, screened on role and industry
- Locations
- Canada, UK, Sweden, Netherlands, Italy, Poland, United States
- Roles
- Consulting, engineering, IT, and operations across software, government, consulting, healthcare, and transport
What we found
Suggestive prompts did the onboarding
First-time users leaned on the prompts to learn how to ask. Several asked for pricing to be surfaced there too, and four wanted product comparisons.
Hover-to-open felt forced
The chat opening on hover read as intrusive. Users wanted the window to appear only when they chose to click.
Privacy was buried
Only three of twelve located the AI privacy statement. Most only cared about it in the context of signing in — but it still needs to be findable.
Icons and overflow menu misread
The chat history icon was routinely confused with restart, and the overflow menu went almost entirely unnoticed.
Users want it to go further
Requests clustered around entering their own scenarios, comparing products, checking how features work with AWS or Oracle, and booking a demo in-chat.
“Overall, how would you rate your experience using this AI chat?”
- Excellent5 · 42%
- Good6 · 50%
- Neutral1 · 8%
- Poor0 · 0%
- Very poor0 · 0%
No one rated the experience poor or very poor — the concierge was perceived as effective and valuable, with clear room to sharpen.
In their words
“I love having suggestive prompts — they provide hand-holding for first-time chat users on how to ask questions.”
“Would recommend making the chat open only when clicked; right now it pops up out of the blue, which seems forced.”
“I want to add a specific scenario from my organization and have the chat help me with that use case.”
“This will help me find the information quickly. It's great!”
What I recommended
- 01Switch the chat icon to click-to-open rather than hover-to-open.
- 02Replace the chat history icon — it reads as restart.
- 03Remove the overflow menu; discoverability and engagement are both low.
- 04Surface the AI privacy statement outside the chat icon.
- 05State chat capabilities and limits upfront to set expectations.
Caveats
Open-ended and follow-up questions couldn't be tested this round because of mockup limits. Findings cover the suggestive prompts on the Watson data integration and observability page only — prompts differ on other product pages.
What's coming up
Refine the experience against the usability and discoverability gaps found here.
Re-test on the live version of the concierge in a real-world environment.
Define and execute Q2 priorities around the highest-impact fixes.
Explore a persona study to align the chat with distinct user needs.