Case study
Streamlining analytics with log delivery for a B2B SaaS product in the CMS industry
I led the UX and product design for a project focused on streamlining Analytics for a B2B SaaS company in the CMS industry. This company offers an innovative headless CMS, or Content Operating System, that enables businesses to create flexible, dynamic digital experiences across various platforms and audiences, with real-time collaboration and composable capabilities.
The primary goal of the project was to enhance user insights for Enterprise customers, reduce costs and support requests by simplifying the analytics experience and promoting the use of advanced features. Through strategic redirection and agile adaptation, our team pivoted from an analytics feature to a streamlined Log Delivery solution.
Key results within 1 month from launch
84% adoption rate
78.4% satisfaction rate
50% reduction of support queries
Methods
User research + Data analysis
Journey mapping + Task analysis
Wireframing + Prototyping
UI design
Moderated and unmoderated usability testing (including RITE – Rapid Iterative Testing and Evaluation)
Collaboration & Timeline
I collaborated with a Product Manager and a squad of Engineers (Engineering Manager, 2 Front-End Engineers & 2 Back-End Engineers). The main stakeholder was the CEO (interim Head of Product).
The project spanned ~2 months and required us to pivot from our initial solution.
This case study has been anonymised for confidentiality purposes. The name of the company and product has been omitted while maintaining the integrity of the project outcomes.
The status quo
The CMS platform's main strength had always been in content creation and composable deployment, but as of late 2023 its analytics and data visualisation capabilities were still lacking, leaving customers with superficial understandings of their deployments.
Recognising this gap, we knew how important it was to offer Enterprise customers advanced visualisation, analysis and optimisation tools. By providing in-depth insights into content workflows and performance metrics, we aimed to boost the platform's competitiveness in the market, reduce costs and support needs.
The initial vision
Joining the team mid-project, I was met with set objectives and a sense of urgency driven by 3 prior changes in leadership and direction. Stakeholders were keen on immediate results, urging us to equip Enterprise customers with tools for swift issue resolution, deployment optimisation and risk prevention to optimise costs and avoid exceeding quotas. I jumped into analysing past research, performed a competitive analysis and started brainstorming a UX solution.
The focus was on tackling unexpected bandwidth spikes and optimising traffic shape for API, API CDN and assets: equipped with enough insights, customers could investigate and rectify anomalies quickly.
Our goal was to reduce the average time to solution from 8-40 hours, with extensive customer support involvement, to 5-60 minutes unassisted, while also consistently decreasing support queries and overages. This would save costs for the company and for the customers as well.
After several rounds of feedback and iterations, we had a solution that included alerts for spikes and detailed visualisation tools that would help for our scenarios and personas. I began validating a low-fidelity prototype with 8 Enterprise customers and 5 internal stakeholders from the Support team.
A strategic redirection
During validation, I found that several Enterprise customers were already using advanced analytics tools, and would not find much value in the proposed solution, despite getting advanced insights.
The team acknowledged the significance of this finding, although unexpected – after discussing assumptions, constraints and other solutions, we strongly advocated for a shift in strategy.
This was risky, as we had already spent 6 weeks iterating on the initial solution and time was tight.
“I think this would be a little bit useful, but it doesn’t solve a huge problem we have such as needing to hook into other third party tools like DataDog.”
The new solution
I teamed up with the Lead Engineer to explore our latest solution which would allow us to leverage third-party integrations: Log Delivery. The feature would seamlessly integrate our detailed logs to customers’ analytics tools, bridging the gap, and we would offer self-serve customers the ability to download logs.
In a week, we run a rapid research experiment, providing logs to customers with bandwidth issues to validate assumptions and key needs: data comprehension, data processing tools, data dump usability and documentation needs. The results were positive, with a value rating of 4 out of 5. Customers were able to efficiently integrate logs to their preferred tools, and get the information they needed seamlessly.
Navigating complexity
Convincing leadership and stakeholders of the value of our Log Delivery strategy proved challenging amidst so many changes in direction, leaving the team feeling disempowered and adrift. Nonetheless, we persisted, advocating for a necessary shift based on insights and our most recent rapid experiment, and ultimately our efforts paid off!
We ensured compatibility with several third-party analytics tools, leveraging Google Cloud Storage Buckets. Our primary focus was on delivering a simple user experience initially, with room for iteration and improvement post-launch – the UI was pretty simple and straightforward as well.
As our customers put it, a faster solution was better than having no visibility at all.
The results
Within 1 month after launch we observed:
84% adoption rate, underscoring a strong market demand from Enterprise Customers
78.4% Customer Satisfaction Score (CSAT)
50% reduction in customer support queries
We implemented tracking metrics for project utilisation, report frequency, undownloaded reports, error reasons and pre-usage support requests volume.
Next steps: improve guides and documentation, gather feedback, plan a second iteration based on metrics and feedback.
Takeaways
This project highlighted the fluid nature of product development and the significance of leveraging research insights at the right time. Yes, we faced unexpected changes in leadership and direction; but we stayed adaptable and lean.
As a team we emerged from this experience stronger, recognising the value of sticking together and trusting our intuition. We learned the importance of avoiding the sunk cost fallacy, reminding us to prioritise outcomes over past investments and to pivot when necessary.
Check out the user interface designs and UX 👇
Log Delivery: setting up and testing the configuration
Log Delivery: connection feedback
Log Delivery: connection successful
Log Delivery: self serve options