How did a T-Rex look alive? From fossil records, we can learn the average size and shape of the species and approximately when they lived. But if I asked you to draw one, you would rely on your imagination. Your picture would be influenced by other artists whose ideas you have seen. But what colour was a T-Rex? Maybe it had feathers!? We don’t really know.
Quantitative data is the skeleton of the user experience. It tells us the size, the type, and the severity of user problems. Qualitative data is the skin and muscle (and maybe feathers) that brings it to life. It tells us how and why. To create a product that works for real users, we need to have both.
If you have been following our PUX news, you might remember our feature about the Tutti.ch Ticino research project in August 2019. The task was to understand why the number of entering visits were going down in Ticino relative to the rest of Switzerland. This was important to understand because Ticino makes up about 40% of product revenue for tutti.ch. The number of entering visits (= number of visits consisting of more than one click) was in decline since 2018, but growing in D-CH. PUX was asked to find out why.
When you notice a quantitative pattern like this one, you first need to figure out why this is happening. If you read the Intro text of this PUXletter astutely, you should know that “why” questions are answered with qualitative research, such as interviews. So the first suggestion from the tutti.ch team was that we could talk to users in Ticino to figure out what was going on. But two things about this problem made us decide to start with a quantitative approach:
1) A KPI like entering visits is very broad and we needed to know where to focus. If you sit down with a few users and ask them to tell you about their experiences with tutti.ch, you will come up with a lot of ideas for what might be going wrong, but no certainty that one of them is the correct (or highest-impact) explanation.
2) We already had a handy comparison group - D-CH - so we could systematically look at different metrics and identify where the problem was (usability? design? appealing inventory?).
So what we did was systematically analyse all of the potential reasons why EVs were declining in Ticino. Was it due to a competitor growing in the region? - No, according to crawled data from TDA, because there was no notable difference between Ticino and the rest of Switzerland. Were there problems with the Italian translations? - Possibly - PUX Design Lead and resident Italian Marta reviewed the site and suggested some improvements. Was it due to a usability problem? An inventory problem? A marketing problem? For these last few questions, we selected standardised questionnaires from the academic literature that have been validated as measures for specific areas of interest: a scale to rate usability, a scale to rate product and design attractiveness, and a consumer motivation scale to identify users’ motivations for using a product. We also asked some general questions about what sort of things users bought or sold on tutti.ch.
What we found was this: in most of the scales we used, the users form Ticino actually gave significantly higher ratings. They also used tutti.ch more frequently compared to the participants we surveyed from D-CH, which led us to speculate that Ticino-based users who are less “loyal” are harder to reach - or that users in ticino either use tutti intensively or not at all.
There was a notable exception to this pattern: Ticino users gave significantly lower ratings on one particular sub-scale of the usability questionnaire meant to measure the efficiency of the system. And this finding went hand-in-hand with another pattern: users from Ticino reported using tutti-ch much more frequently for the “verticals” Real Estate, Cars, Jobs, etc… and much less frequently for second-hand marketplace items compared to the D-CH users.
These verticals have unique usability requirements - especially around search - that tutti.ch was not specialised for. Intuitively, it made sense that users trying to find a house would consider the system less efficient than users looking for a used rug. Such usability issues, over time, can lead to a gradual user churn.
There were some other potential factors for the EV decline based on our analyses, but the use case / usability mismatch between Ticino and D-CH was the most obvious finding. We recommended that tutti.ch should treat Ticino-based users differently to D-CH users - making sure to conduct usability tests in italian, for example. And now there was a better idea of where to focus to answer the actual why questions. This is where we handed back over to the tutti.ch research team.
Since then, the tutti.ch team has conducted qualitative follow-up research to get a comprehensive understanding of the user needs and expectations while searching for a new home. In-person interviews and a “journey mapping” task moreover revealed crucial contextual aspects that influence users’ search behaviour, like the competitiveness of the local real estate market. For example we learnt that people looking for flats in cities like Zurich and Basel are very reluctant to use any filters that restrict their search too much. They fear to miss out on their dream flat in the highly competitive market.
In the next steps we ideated with the whole project team, including Marta from PUX, on features that will solve the identified user needs in an ideal world... and prototyped the prioritised features, before releasing the first iteration of the improved real estate user experience.
Monitoring our success metrics we quickly noticed that neither sessions nor ad replies were improving. While we didn't expect these metrics to instantly improve after the release, over time and in combination with the qualitative feedback collected over tutti.ch’s Happiness tracking Survey (HaTS) and user comments, this indicated dissatisfaction and pain points of our users.
So one more time we set out to understand the dinosaur skeleton: What is the size, type and severity of those pain points people reported in the HaTS? (see also the featured project on HaTS @ CAR FOR YOU by PUX) We set up a pop-up survey on the webpage to “catch” people in the context of searching for real estate on tutti.ch. We were startled to see that the two biggest issues revolve around the search scope: people either don’t know how to further refine to get to relevant results or arrive at not enough results in the first place.
To add the “feathers”, we observed users’ behaviour in user tests and held interviews. How do people go about searching the flat with the new real estate XP? Why do people feel the search scope is not right? In Ticino in particular, we found that people missed the more fine granular area search that was replaced by the new location search. With this knowledge we could re-introduce the area search whilst still allowing the new radius search.
Since tutti.ch implemented the changes to the real estate search experience, we saw a significant increase in user happiness with real estate search among Italian-speaking users, as measured by their continually running HaTS survey! We will continue to monitor these numbers to confirm whether the changes make an impact on usability ratings - and of course on EVs.
In the monthly Product and UX letter the PUX team shares insights about projects, interviews with product gurus and personal anecdotes inside and outside the TX group: