In the spring of 2020 the world had to quickly adapt to the new situation of COVID-19 spreading across the globe and causing lockdowns. The situation for us at Kaiku, was no different. The development of our product naturally needed to continue just like before – or even at a faster phase, as it could be used in the follow-up of the COVID-19 as well as it proved to be even more essential in the routine care of patients, who were risk groups for the disease and thus advised to stay home and isolated.
At Kaiku, usability testing is a core activity in our product development cycle, and it is performed with both our patient and HCP users. It is indeed an important part of product development, as it helps in creating more usable products by involving the actual users and getting their perspective and feedback as well as helps to meet set usability objectives [1][2][3]. Usable products, then, can forward the adoption of products as well as create customer loyalty [4]. To clarify, with usability testing I am talking about the activity focusing on observing users that are working or interacting with a product and performing real and meaningful tasks [5].
Before this spring, usability testing at Kaiku was done mostly locally, often at our client clinics’ facilities, to reach our users in an effortless way. After the start of the pandemic, it was soon realized that new practices are required, and a remote usability testing process should be implemented in order to continue usability testing during this unusual time. Remote usability testing would not only be needed during the pandemic but also later on as the company grows and reaches new markets. As this need became self-evident, I then realized that this was something to study and do my thesis about.
As the HCP users have been more accessible to us, for example during clinic visits, the focus of my thesis was set to our patient users. Hence, the interest was also on synchronous remote usability testing, meaning usability testing where both the organizer(s) and the participant would attend the test session at the same time but from different locations – this would allow the organizers to for example help the participant with setting up the test environment and also in possible issues during the test. So, there my new thesis topic was: gathering experiences on remote synchronous usability testing with patients, which would then help the company in creating a proper testing process for the future.
I started my thesis project by studying the existing literature and from there finding the best recommendations, requirements, and guidelines for both remote synchronous usability testing as well as usability testing with patients and/or older adults. These recommendations were many, and they included guidelines for example for handling the participants’ data, easing possible cognitive overload with written instructions, keeping in mind the participants’ different skills or lack of those, conducting a pilot, and tips for using a prototype.
Based on the findings that I was able to gather from the literature, we created a usability testing process together with my colleague, an experienced UX designer, who had created a prototype to be tested and with whom I was then also executing these usability tests. The usability tests were measuring the usability of an improved symptom questionnaire and they were executed on the 11 participants’ mobile devices and by having simultaneously a video call open throughout the sessions. Along with testing the prototype and following the users’ actions, we also utilized multiple data-gathering methods like observation, semi-structured interviews, and the system usability scale (SUS). With these methods, we were able to gather a lot of data that was then rearranged and sorted by data coding and grouping. By the end of the data analysis, the codes from the gathered transcriptions were divided into two categories: codes related to the usability of the prototype or the Kaiku service or codes related to the remote usability test session, its tools, and practices.
Screenshots of the prototype used in the usability tests
The usability tests really helped to improve the new symptom questionnaire and they also provided us extremely valuable feedback regarding the product in general. Since the objective of my thesis was to gather experiences of remote usability testing with patients, I was, however, more interested in the latter of the two aforementioned categories. From the gathered data, I was finally able to compose my results regarding the key learnings and my recommendations regarding remote synchronous usability testing.
The results altogether 26 key findings or recommendations from my empirical study were divided into 7 different categories (see graph below), which were created based on the different sections of the actual usability tests. The recommendations mentioned for example the importance of clear and consistent communication in sent messages and materials, tips for when using a prototype in a usability test, factors to consider when choosing a prototyping tool as well as general recommendations for organizing remote usability test sessions. I concluded my thesis by comparing the results from the literature and empirical studies and the results were mostly similar and supported each other: both of these highlighted for example the importance of written instructions to ease possible cognitive overload, keeping in mind the participants’ age and possible symptoms when planning the sessions, and having multiple test organizers present in actual test sessions.
Although the year has been quite extraordinary, and not in the best possible way, the thesis project was really insightful for me. Not only did I have the chance to create a remote usability testing process and develop it further by gaining real experiences, but I also got to spend time with our users and learn from them. Although my thesis is limited to only one study for one company, I hope that the results are still relevant not only for Kaiku and my colleagues performing remote synchronous usability tests later on but also to others doing so in different projects and organizations.
Katri Saarinen
Software Engineer & part-time UX Designer
If you are interested in diving deeper into my thesis and its results please contact me at katri.saarinen@kaikuhealth.com
References
- Thompson, K. E., Rozanski, E. P., & Haake, A. R. (2004). Here, there, anywhere: Remote usability testing that works, In Proceedings of the 5th conference on Information technology education.
- Thompson, S. (2003). Remote observation strategies for usability testing. Information Technology and Libraries, 22(1), 31.
- Lewis, J. R. (2006). Usability Testing (tech. rep.). IBM Software Group. http://drjim.0catch. com.
- Kortum, P., & Peres, C. (2015). Evaluation of Home Health Care Devices: Remote Usability Assessment. JMIR Human Factors, 2(1), 9. https://doi.org/10.2196/humanfactors.4570
- Barnum, C. M. (2020). Usability testing essentials: ready, set… test! Morgan Kaufmann.
