Insider is a Sequoia-backed unicorn and B2B SaaS platform offering marketing technology solutions for leading global businesses.
My Role: Since joining Insider in 2023, I have been responsible for the design research process for one product area and a cross-functional product team. I collaborate closely with the UX Research, Product Design, Product Management, Development, Partner Success, and Data Warehouse teams.
This is an evaluative research study I led at Insider. The high-level objective was to understand users' needs and expectations regarding Smart Journey Creator in Architect (Insider’s Journey Builder) and how effectively it meets users’ needs. This feature enables digital marketers to create omnichannel marketing automation flows powered by Insider’s AI solution, Sirius AI™.
Disclaimer: Certain details and visuals on this page have been removed or adjusted due to confidentiality. The images included are from the final report and serve to showcase my research reporting skills; they may differ from the actual outcomes.
Smart Journey Creator in Architect: See it in action.
Process
Background & Goals 🎯
After the release of the beta version of the Smart Journey Creator feature in Architect, initial user feedback we received suggested that users might not be fully aware of its capabilities and limitations. Additionally, we observed low adoption rates, with most users discontinuing their use after their initial trial. Therefore, we decided to undertake a user research study to investigate these issues further.
Our main aim was to gather rich qualitative insights into users' interactions with the feature, identifying existing usability issues and pain points. We also explored users' needs and expectations for this feature. The ultimate goal was to improve feature usability and adoption while reducing time-to-value (TTV).
Research Questions 🔍
RQ1: How do users interact with the Smart Journey Creator feature?
RQ2: What are the pain points, needs, expectations, objectives and use behavior of users when using the the feature? What are the intervention areas for design?
RQ3: What are the differences in terms of needs, expectations, and use behavior/patterns regarding Smart Journey Creator between the identified user segments?
Methodology 💼
1. Clickstream analysis via FullStory: The goal of this step was to investigate the key usage metrics of the feature (i.e., top users, number of sessions, session duration, rage clicks, prompts they entered, and their flows).
2. Moderated Usability Testing with beta feature: The goal of this step was to uncover users' pain points, needs, and expectations while using the Smart Journey Creator, explore their interactions with it, and measure their attitudes toward adopting this AI-based solution.
Sampling Criteria 👤
We defined two sub-groups based on our assumption that users' expectations of the feature, and consequently their experience with it, would differ depending on their experience levels:
1. Experienced users: Enterprise-level customers who have used the Smart Journey Creator feature at least once and with good Architect knowledge/proficiency based on customer age/lifetime & panel usage.
2. New users: Newly onboarded users who have not used the feature yet and with limited product knowledge.
I created a FullStory dashboard to monitor feature usage metrics and analyzed user prompts submitted to the bot. I then conducted moderated usability testing with 10 participants from two user groups.
Results
Key Insights from the Study 💡
Insight #1: Guidance 🧭
Providing step-by-step guidance during prompt entry can help users achieve their desired outcomes more effectively.
Challenge ⚔️
I found out that;
● Users struggle to interact with the Smart Journey Creator due to insufficient guidance on writing prompts.
● Users feel confused and frustrated when receiving unexpected or erroneous results, as they don’t know how to refine their prompts to better align with their goals and the current capabilities of the feature.
Recommendation💡
Introduce step-by-step guidance on input expectations to help users craft effective prompts while maintaining flexibility for users to describe their intentions freely.
Insight #2: Reasoning Behind Results 🧠
Providing reasoning behind the generated results to support the reliability of the feature and help users make sense of the result
Challenge ⚔️
I found out that;
● Users need to understand the justification/reasoning behind the generated journey to determine how to use it in accordance with their goals.
Recommendation💡
● Providing a brief explanation of the generated journeys, as to based on which criteria the journey logic is created (e.g., based on your database/most used channels), can help users to better make sense of the results and increase the perceived reliability of the feature.
● A reasoning behind the results is often provided on some AI tools such as Chat GPT.
Some key insights from the final UXR report (the rest is blurred due to confidentiality).
Code frequencies from the qualitative data analysis (numbers adjusted for confidentiality).
Prioritization & Ideation 📊
Based on the study findings, I created user stories to help the team prioritize tasks. These were ranked by issue frequency, importance from a UX POV, and the effort required for development.
Research findings prioritization sheet.
Research Impact 🏆
Strategic & Product Impact:
● The research guided product development and informed key feature improvements.
● Feature adoption increased by 42% within three months after implementation.
● Product designers, developers, and product managers successfully incorporated actionable insights from the research into the product, enhancing its usability and overall user satisfaction.
