This is an evaluative research study I led at Insider. The high-level objective was to understand users' needs and expectations regarding Smart Journey Creator in Architect (Insider’s Journey Builder) and how effectively it meets users’ needs. This feature enables digital marketers to create omnichannel marketing automation flows powered by Insider’s AI solution, Sirius AI™
About Insider 🦄
Insider is a Sequoia-backed unicorn and B2B SaaS platform offering marketing technology solutions for leading global businesses.
My Role: Since joining Insider in 2023, I have been responsible for the design research process for one product area and a cross-functional product team. I collaborate closely with the UX Research, Product Design, Product Management, Development, Partner Success, and Data Warehouse teams.
Disclaimer: Certain details and visuals on this page have been removed or adjusted due to confidentiality. The images included are from the final report and serve to showcase my research reporting skills; they may differ from the actual outcomes.
Smart Journey Creator in Architect: See it in action.
Process
Background & Goals 🎯
After the release of the beta version of the Smart Journey Creator feature in Architect, the initial user feedback we received suggested that users might not be fully aware of its capabilities and limitations. Additionally, we observed low adoption rates, with most users discontinuing their use after their initial trial. Therefore, we decided to undertake a user research study to investigate these issues further.
Our main aim was to gather rich qualitative insights into users' interactions with the feature, identifying existing usability issues and pain points. We also explored users' needs and expectations for this feature. The ultimate goal was to improve feature usability and adoption while reducing time-to-value (TTV).
Research Questions 🔍
RQ1: How do users interact with the Smart Journey Creator feature?
RQ2: What are the pain points, needs, expectations, objectives, and use behavior of users when using the feature? What are the intervention areas for design?
RQ3: What are the differences in terms of needs, expectations, and use behavior/patterns regarding Smart Journey Creator between the identified user segments?
Methodology 💼
1. Clickstream analysis via FullStory 📊 The goal of this step was to investigate the key usage metrics of the feature (i.e., top users, number of sessions, session duration, rage clicks, prompts they entered, and their flows).
2. Moderated Usability Testing with beta feature 🧪 The goal of this step was to uncover users' pain points, needs, and expectations while using the Smart Journey Creator, explore their interactions with it, and measure their attitudes toward adopting this AI-based solution.
Sampling Criteria 👤
We defined two sub-groups based on our assumption that users' expectations of the feature, and consequently their experience with it, would differ depending on their experience levels:
1. Experienced users: Enterprise-level customers who have used the Smart Journey Creator feature at least once and with good Architect knowledge/proficiency based on customer age/lifetime & panel usage.
2. New users: Newly onboarded users who have not used the feature yet and with limited product knowledge.
I created a FullStory dashboard to monitor feature usage metrics and conducted a log analysis on user prompts submitted to the bot. I then conducted moderated usability tests with 10 participants from two user groups.
Tasks 📝
I designed the task to observe how participants would interact with the feature in a real-life context. If the participant couldn't imagine a use case during the test, I've given them a pre-defined use case.
"Open your panel and take a look at the recent campaigns that you’ve created on Architect. Select one of the use cases/campaign ideas and see how you can go about creating a similar campaign by using the Smart Journey Creator."
A journey generated by the Smart Journey Creator based on a user prompt. This screenshot is from the new version of the feature, enhanced based on the findings and insights from this study.
Results
Key Insights from the Study 💡
Insight #1: Guidance 🧭
Providing step-by-step guidance during prompt entry can help users achieve their desired outcomes more effectively.
Challenge ⚔️
I found out that;
● Users struggle to interact with the Smart Journey Creator due to insufficient guidance on writing prompts.
● Users feel confused and frustrated when receiving unexpected or erroneous results, as they don’t know how to refine their prompts to better align with their goals and the current capabilities of the feature.
Recommendation💡
Introduce step-by-step guidance on input expectations to help users craft effective prompts while maintaining flexibility for users to describe their intentions freely.
Insight #2: Reasoning Behind Results 🧠
Providing reasoning behind the generated results to support the reliability of the feature and help users make sense of the result.
Challenge ⚔️
I found out that;
● Users need to understand the justification/reasoning behind the generated journey to determine how to use it in accordance with their goals.
Recommendation💡
● Providing a brief explanation of the generated journeys, as to which criteria the journey logic is created (e.g., based on your database/most used channels), can help users to better make sense of the results and increase the perceived reliability of the feature.
● A reasoning behind the results is often provided on some AI tools, such as ChatGPT.
Some key insights from the final UXR report (the rest is blurred due to confidentiality).
Code frequencies from the qualitative data analysis (numbers adjusted for confidentiality).
Prioritization & Ideation 📊
Based on the study findings, I created user stories to help the team prioritize tasks. These were ranked by issue frequency, importance from a UX POV, and the effort required for development.
Research findings prioritization sheet.
Research Impact 🏆
Strategic & Product Impact
● The research guided product development and informed key feature improvements 🧭
● Feature adoption increased by 42% within three months after implementation 📈
● Product designers, developers, and product managers successfully incorporated actionable insights from the research into the product, enhancing its usability and overall user satisfaction 🚀
