Using evaluative UX Research to assess and inform design decisions

As one of Canada’s largest and most beloved media organizations, the CBC produces a huge variety of news, radio and television programming. Their website also includes a music subdomain which offers a wide variety of music-related content, including news, reviews and the ability to stream songs and radio shows.

In 2014, the CBC team hired Toronto-based, UX strategy & design research firm Usability Matters to assess the impact and usability of their most recent redesign. Updates included changes to global navigation, the introduction of new features, such as a music finder (aka “streambox”) and the ability to save and share playlists.  To accomplish this assessment, the Usability Matters team recommended that we conduct a remote, moderated usability study. This would allow music lovers to give us valuable feedback and ensure that any changes the CBC team made were user-centred and data driven.

In my role as User Experience Designer, I planned and conducted (my first!) moderated usability study, reported insights and made interaction design recommendations.  I was responsible for several deliverables, including:

Defining the Research Objectives

In order to better understand the project’s strategic objectives and the site’s current state, I reviewed the client’s existing primary research and performed a heuristic analysis of the website. These activities informed the research goals which included assessing participant’s wayfinding behaviour, the affordance of icons and the usability of new features (such as the music “streambox” and playlist features).

Drafting the Test Script

I created the script by defining tasks which would compliment the research objectives.  Since the scope of the test was fairly large, we “tested the test” first to ensure tasks could be completed within an hour. This also allowed us to assess the script’s flow and wording. By including open-ended questions, we ensured that natural behaviour could be observed – and research objectives could be met – without leading participants.

Facilitating Remote Testing Sessions

In the days before usability testing platforms made it easier to connect with users anywhere, remote testing was  much more technically challenging to execute. For example, we had to ensure that we could capture (and edit) a quality recording of the participant’s behaviour. Given that our group of participants were using a variety of devices, we also had to be prepared to solve unpredictable technical glitches. This extra effort was worthwhile because it allowed participants to perform the test in a familiar environment, resulting in more reliable observations.

Reporting Findings & Recommendations

 I drafted and presented a report which presented our findings and interaction design recommendations in a concise and visual way. To ensure the findings had impact, I included screen shots and video clips from the remote sessions. 

Test results indicated that participants were able to create playlists successfully, but there were some issues with certain navigation labels and icons. The study also revealed that the participants struggled with the new “streambox” on desktop devices (since it was a design pattern that borrowed from mobile design but was unfamiliar to desktop users). This showed the CBC team the power of mental models; users were not successfully associating a mobile pattern with a desktop experience and this was leading to confusion and way finding issues. This insight led to the CBC team rethinking the “streambox” as an approach to content discovery. 

This project was one my first introductions to a formal UX research methodology and, among other things, it showed me the impact that user-centred research can have to inform decision-making.  Since then, I’ve conducted or consulted on hundreds of usability tests.

This project also allowed me to discover how much I love helping teams generate rich insights by defining relevant research goals, determining an effective task sequence and crafting meaningful scripts. If you’d like to chat more about evaluative research and UX strategy – or if you’d like to see more detailed samples of my work on this project, please get in touch!

Check out the CBC Music website:

Go to the CBC