Generative AI is popping up everywhere, in every sector – even design research, so we had to try it out. Synthetic Users is an AI tool that provides qualitative interviews with (yes, you guessed it) synthetic, AI-generated users. The site says it is able to interview users, explore design challenges, send out surveys, identify problem spaces, give feedback on design ideas, test assumptions, run a brainstorm session, analyse competitors, and more. It sounds like it can do a lot of what we do, so we wanted to put it to the test by running the same study with real users and synthetic users – here’s what happened!
We wanted to put it to the test by running the same study with real users and synthetic users
Running a study about AI using AI
We just finished up a client project, consisting of three workshops and multiple one-on-one interviews, about people’s experiences with using and integrating AI at work. We thought this would be a good, open problem space to test with synthetic users, plus we already had lots of (real) qualitative data and a full report to compare with.
After logging in, we landed on the Synthetic Users dashboard. We chose the option to run a dynamic interview directed by our research goals to explore the problem space of how AI is used at work and workers experiences with this. We set up the research as similar as possible to the real client project (without sharing any real project data, of course).
We generated six synthetic users (our ‘participants’ as they call them). We tweaked our participant profiles (age, profession, attitude towards AI, etc.) to reflect the spread in our sample of real humans. Happy with our participants, it was time for the magic!
The loading screen read: ‘Identifying potential interviewees and checking availability … Scheduling interviews … Conducting interviews, taking detailed notes … Thanking participants…’ The steps that typically take us weeks happened in a matter of seconds. The designers behind the system definitely have experience with qualitative research to think about including all of these stages in writing, but it does feel kind of dystopian that these delicate research activities pass by so quickly.
Identifying potential interviewees and checking availability … Scheduling interviews … Conducting interviews, taking detailed notes … Thanking participants…
The system then spits out organised interview transcripts from each participant and an overall summary. We analysed each document and compared it with the insights that arose from the real research study. Largely, the themes and experiences mentioned by real and synthetic users were similar, but some things were definitely missed.
How real can fake users be?
In cross-comparing the data, we found some stark differences across the two groups of real and synthetic users:
- Real users speak with more emotion and use language like “I feel” or “I think”
- Real users are often unsure of exactly what they think and can sometimes change their stance on a topic within the session – this is a messy fact of life
- Real users’ responses are based on specific experiences that they’ve had, like “this one time with a colleague…” or “last week, this happened…”
- Synthetic users’ responses are hyper-analysed and express “emotions” in a way that doesn’t feel natural or vulnerable
- Synthetic users don’t allow space for researchers to read between the lines – the experience and their reflection on it are already put together in the same paragraph
- Synthetic users come across as too articulate and tend to be solution-focused – there no room for unsureness in an answer
Synthetic user: “The legal profession is losing its human touch in favour of efficiency and automation, is deeply unsettling. Coupled with this anxiety is a feeling of isolation.” (they are talking about emotions but not using “I” statements and have the advanced emotional literacy to identify anxiety with isolation.)
Real person: “I sometimes feel like I don’t think as much anymore and may be relying on the AI to do my job at times.” (They use reflective “I” statements, like they are realising this for the first time in the moment and are unsure if their AI usage is too much at times.)
What this taught us about being human
Humans are messy, unpredictable and emotional beings. That’s why we love doing this work. We get to listen to people’s stories, get to know what’s important to them and then craft many different people’s stories into insights for design teams to be inspired by. In this way, we bring messy, real life experiences to the forefront – and often in these everyday, recognisable dilemmas, we find the most interesting insights pointing to opportunity spaces for better support.
AI, in essence, is the opposite of messy, unpredictable (well, maybe with the exception of hallucinations) and emotional. AI is designed to be structured, concise and emotionless. These are not bad things, but it’s good to think about as a qualitative researcher. This level of structure can be beneficial in other parts of the research process. For example, AI can help initially identify relevant topics to explore with participants, can help give structure to an analysis framework or setting up a report, etc. Our work is a delicate balance of being unstructured and structured. And maybe a tool like this can help us strike that balance even better.
All of the participants (real and fake) in this research shared their stories with us about how they use AI in their jobs, so it felt only right to share ours, too. This trial strengthened the fact that we need to continue valuing real human stories to inform products, services and experiences that real humans will interact with. Designs are meant to fit into and enhance people’s lives, and we must remember how messy, unpredictable and emotional real life can be in order to design for these realities. It will be interesting to see how AI tools will evolve in this sense. We’re keen to keep following this.
We must remember how messy, unpredictable and emotional real life can be in order to design for these realities
By Katy Barnard & Evelijn van Hilten
Images from syntheticusers.com