This user research project revolves around an internal contracting service that helps procure talent across the Ontario Public Service (OPS). A new vendor management system was recently implemented to support all requests' life cycle and reporting. At the Ontario Digital Service, our role was to conduct an internal discovery research project to shed light on the current user experience and help inform future decisions on improving the service for the staff.
There were some research restrictions due to circumstances outside of our control. We only had access to interviewing internally facing user groups. We could not access the vendor management system, so we could not see visually how it works.
Research limitations can seriously hinder a research project depending on how we view it. We crafted our research guide to ensure all possible avenues were covered. I leaned on my facilitation skills to uncover as many insights as possible. We also emphasized limitations in our final report and the opportunity to do further research in the future.
This user research project began in the winter of 2023. The team started the user research plan, but it had to be paused for reasons outside their control. In May of 2023, I joined and resumed the work where they had left off. My initial responsibilities included reviewing the project background, finalizing the user research plan, and preparing facilitation guides and screener surveys.
We would have preferred to conduct usability testing to better understand how users interact with the vendor management system. However, since we didn't have access, we opted for 1:1 semi-structured interviews. We also chose not to use focus groups to avoid conformity bias.
This was the first project at the Ontario Digital Service (ODS), and I was fortunate to be given the opportunity to take the lead on the recruitment process as well as facilitate almost all of the interviews.
We finalized the user research plan with our partners and began recruiting. During our weekly check-ins with the partners, we discussed our need for contact information, and they provided us with over 800 email addresses. My responsibilities included creating email templates, contacting user groups, and scheduling interview sessions.
Perhaps it was a busy time for this user group, resulting in 15+ reschedules or no-shows and the 22/800 screener response rate. To counter this challenge and ensure timely in-scope delivery, I set up timely follow-up emails while keeping an eye on the mailbox to quickly accommodate rescheduling.
Flow state is a mental state when someone is fully immersed in using a combination of their skills while being challenged in a task or activity. Understanding that I would be facilitating most of these interviews, I sought feedback from management and colleagues to improve my skills on the go.
Reflecting on the interviews, we found that the user groups were closely related, so the intern and I set up several synchronous meeting sessions to ensure proper and consistent analysis alignment. We chose affinity mapping to uncover broad themes from a significant amount of data. After filling any gaps in the interview notes, The intern and I began affinity mapping the 5 user groups one user group at a time. This process helped us condense 22 interview frames into 5 affinity maps.
To analyze the user data effectively, we ran a thorough comparison of the five user groups. Through this process, we identified common themes and insights across the groups, which we have compiled and presented in a separate section named global findings.
As time passed, there became more global themes than unique themes for user groups. I learned that this can sometimes happen when a service and its users' roles overlap, and that's okay. We took this understanding to inform out report writing approach.
For this user research project, our job was to shed light on the current user experience through insight and recommendations to inform strategic decisions moving forward. My responsibilities at this stage included compiling our insights into a cohesive report presentation for the lead to craft the recommendations.
Sometimes, structuring the report according to the service steps can be effective. Since our findings became almost all global themes, we structured our report based on those. Further, the partners were familiar with the process, so they could digest the information this way.
Due to restrictions outside our control, we could not see the product in action. This restriction meant we could only paint the picture using participant feedback in our interviews. That being said, the recommendations were a combination of broad and specific. It was an excellent opportunity to learn how to craft recommendations with a variable lens.
At the beginning of the presentation, we asked the partners if they would like a formal or more conversational presentation style. They picked the latter, and we worked together to figure out potential solutions and next steps, which I don't think we would have gotten if we presented more formally.
Early on, when starting out moderation, I always tried to find the next follow-up question so there would be no awkward silence. One time, I couldn't find a follow-up question after the participant finished their thought. They elaborated on their point when I thought things would get uncomfortable!
I've learned that something always comes up for recruitment, no matter how much we prepare. If possible, adding a week or two to the recruitment timeline is a safe bet.
This user group has had a little over a year with this new vendor management system, so they have a lot of thoughts built over time. Some participants even had notes prepared. In one interview, I asked maybe five questions in total. So long as things are relevant, it’s okay to go off-script!
