Uxability testing to reduce risks | Vita Yates

Leveraging usability testing to reduce risks before launching an application in an agile environment

Role

UX Designer and Researcher

Company

BC Public Service

Team

Agile product development team (developers, Sr. UX Designer, Service designer, PO, SM, business analyst)

Skills

Usability testing

Affinity diagrams

Wireframing

UI design

Problem space

The Compliance and Enforcement team was launching a new application for managing public complaints and investigative case files across the Natural Resource Sector for the BC Government. After a successful pilot launch with a small group of users with the Conservation Officer Service, the team was preparing to launch the application across the whole organization of over 200 users.

The application had several minor usability and cosmetic issues that had not been addressed by the team. Additionally, we had not had the opportunity to talk directly with the pilot users. The application was replacing a legacy system and was introducing new business processes. Because this application is critical to the daily operations of the organization, there were large risks of launching the application without gathering feedback from the pilot users.

Process

Usability testing

To reduce risk, I determined that it was critical to conduct usability testing prior to launching the application across the organization. We needed to identify the existing usability issues in the application as well as test new features that were recently released and had no been used during the pilot launch.

Planning

I lead the planning and preparation for this research and involved my team throughout the process to ensure the successful implementation of the research plan.

Purpose and goal

To get feedback on the pilot launch to determine if improvements or additional features are needed before the organization wide launch
Test new features to ensure they meet users needs

Interview format

Each interview included open ended questions and a series of usability testing tasks
I created interview guides tailored to each participant type

Interview script

Intro

Task 1:

Task 2:

Task 3:

Experience with new system?

Biggest issues?

Recruitment and participants

I managed the recruitment of participants. I leveraged existing contacts I have built with the business area as well as recruited new participants who have had less contact with the development team.

The final participants included:

3 Managers
4 Conservation officers
1 Administrator
2 Other

Conservation officers

Managers

Administrator

Other

Timeline and agile development schedule

We had about a month to complete the testing and deliver recommendations to the team for development. This fell over the winter holidays, making scheduling a challenge.

The team worked in two week Agile sprint cycles, so I structured this research around those release deadlines. To ensure the development team had enough time to implement any necessary changes, I took a "just enough research" approach and planned to conduct interviews and do synthesis in tandem.

Conducting interviews

All interviews were conducted virtually. I was the lead facilitator for half the interviews. Each interview also had a note taker and one or two other members from the team. This was to help build buy in among the team and allow others to hear directly from the users.

Notetaking

Because we were on a tight timeline, I set up a virtual whiteboard (MURAL) to take notes directly on. This expedited the synthesis process.

Synthesis and prioritization

I used affinity mapping to identify themes and key pain points. Once the data points were grouped, I worked with the team to prioritize the issues. We focused on the critical usability issues and the quick, low effort wins.

Sharing "wins" with the team

In addition to sharing the key pain points with the team, I also shared the positive feedback we received. Sharing these successes at an internal team meeting and our external sprint review allowed people to hear the positive impact the work is having.

"The team has been listening to everything we have to say and is trying to make it work for us."

Sergeant

"quick close button is awesome"

Officer

map feature really simplifies things so i don't have to manually look for coordinates. that really cuts down my work
tremendously.

Officer

Implementation

I worked closely with the other designers and developers on the team to quickly iterate on solutions to the problems we identified. The focus was on small incremental improvements that could be quickly implemented by the developers over the following 1-2 sprints.

Impact

By bringing real users into the process early, we were able to uncover and address key usability issues before the application launched more widely. This allowed us to make targeted improvements that better supported how people actually work, not just how we assumed they would. The feedback helped us prioritize what mattered most, from fixing confusing screens to improving small interactions that made a big difference. Collaborating across roles - design, development, and business - helped us move quickly and build confidence in the product. In the end, we delivered a system that was more intuitive, more useful, and better prepared for a successful launch.