Like everyone, we at Stby are getting to grips with not only what AI does, but also the impact of AI on people, teams and working together. Here, two of our team, Qin Han and Ed Louch, discuss the hot topic of synthetic users and bring their insights out in the open.
Recent signals
Ed: It feels like every conference we’ve been to lately has one topic on repeat: synthetic users. It’s everywhere. You mentioned hearing a specific example recently?
Qin: Exactly. They are like the new personas. Remember the old-school personas? The ones where you have a static face on a slide and a couple of paragraphs about their demographics and hobbies. Instead, companies are developing internal digital personas for the entire business. It’s essentially an AI agent that anyone in the company can interview.
Ed: So, instead of reading a report, you interview a bot?
Qin: It’s built on existing marketing data and past research. But it raises so many questions about the foundation. If you’re a global company, who decides which data goes into that AI? Is it heavily biased toward English-speaking markets? If you’re using qualitative data to train it, do you have someone with research training supervising that build to ensure it’s not rubbish in, rubbish out?
Reality vs. representation
Ed: That’s the worry, the conclusion the AI draws is only as good as the source. What about using generic tools like GPT-4 to take on a persona for testing?
Qin: I see the appeal, but there’s a certain naivety there. If you use a generic AI, you’re assuming that as long as the bot’s opinion is different from yours, it’s valid. But GPT is trained on the internet, which represents people who have a loud voice on Reddit or Twitter.
Ed: It’s such a filtered version of perspectives.
Qin: That is only one of the problems with using AI trained with generic internet data. Think of your LinkedIn page versus your actual career. Your career is messy, complicated, and full of context. Your LinkedIn is the polished, filtered version. AI only sees the LinkedIn page. It’s not lying, but it’s a specific, flattened version of reality.
Ed: So, it’s fine for a first pass decision, but you wouldn’t hire someone based only on their CV.
Qin: Exactly. If the research question can be answered at a CV level, use the AI. But if you need the nuance, the why behind the messy human experience, you still need to talk to real messy human beings.
The researcher as mediator
Ed: So, what happens when we remove the researcher as the mediator? When does that become dangerous?
Qin: It depends on the risk of the decision. If you’re making small adjustments to a product in a mature market, a synthetic user is a great sense check. But if you’re launching something truly new, you need the messiness of a real person.
I’ll admit, as a researcher, I have a biased view. I want to maintain control! I want to say, “I do the real research, and you’re just playing with a toy.” But we have to move past that. The conversation should be: When do we use both?
Ed: Right, because if researchers aren’t proactively involved, people are going to use these tools anyway, and they’ll likely misuse them. We need to be the ones building the guardrails.
Qin: Absolutely. We need to define parameters where the synthetic user is designed to say, “I don’t know, I don’t have enough data to answer that.” It’s about narrowing the scope so that when we do talk to real humans, we’re asking the most valuable questions possible.
Synthetic users as a research output
Ed: This makes me think, what if synthetic users weren’t just internal tools for big corps, but an actual output of our research? Something we hand over to a client at the end of a project.
Qin: I see it as the evolution of the persona. Because, let’s be honest, how many of those persona decks we deliver actually get used six months later?
Ed: We could deliver essentially interactive digital archetypes that represent the research. We do the deep foundational work, understanding the motivations and fears of people, and then we create synthetic users based on this. Clients would be able to continuously come back to and interact with them, so it becomes a feedback loop. The research doesn’t get filed away and forgotten; it stays alive, acting as a constant, evolving sparring partner for their design decisions.
Qin: I actually wouldn’t be bothered by that at all. If it’s done well, it’s more engaging and precise. I’d much rather a design team use a synthetic version of my research to sense-check an idea than not use the research at all.
Ed: It prevents that rubbish in, rubbish out problem because the model is grounded in high-quality foundational work. And since human motivations don’t change as fast as products do, it remains a valid partner for the client for a much longer time.
Qin: It changes us from being the messengers to being the curators of a living knowledge base. It’s definitely a more plausible use case than it was even two years ago. I’m curious to see which of our clients would be brave enough to try it first.