Usability Testing of PHIgo
A team project that conducted a usability test for PHIgo, a GE Healthcare product that is designed for researchers within the medical field to analyze various medical imaging modalities and make diagnostic decisions.
Type
Team Project
Duration
Sep. - Dec. 2020 (12weeks)
Role
Facilitated client meetings, hosted usability testings (online), worked with the team including conducting heuristic evaluation, planning the study, conducting usability tests, etc.
Team
Rong Zhao
Interaction Designer (MFA Visual Communication Design )
Roshni Wadhwa
UX/UI Designer (MS Human-Computer Interaction)
OVERVIEW
Context
About PHIgo: A new integrated software platform that's based on the principles of real omics and it takes advantage of artificial intelligence-driven imaging processing for analysis of various medical imaging modalities.
Target Users: Researchers within the medical field (mainly Ph.D. students)
Product Lifecycle: Market introduction stage - They released the first version of PHIgo a month before we started the project
Restrictions: Due to the company's Non-disclosure Agreement, we can only get access to the application by remote controlling the client's computer versus installing it on our computers
Test Environment: Online via Zoom (Due to Covid-19 policy and the company's Non-disclosure Agreement)
Goals
Evaluate
Uncover usability issues and assess the overall effectiveness of PHIgo
Capture
Capture user feedback and unmet expectations
Recommend
Provide evaluation report and actional recommendations to improve the user experience
Process
Heuristic Evaluation
We evaluated 3 screens that carried the main features of the PHIgo based on 16 Usability Heuristics.
Screen 1: PHIgo Homepage
Screen 2: Application Center Dashboard
Screen 3: LK page (one of the apps)
Takeaways
The key features we evaluated are “Import” “View” “Add to Apps” and “View in Apps”. Based on our evaluation, most of the problems are related to “External Consistency”. Based on the issues we found, we are going to make a plan to test the ease of understanding different functions of completing a medical imaging study task.
Test Plan Design
Research Questions (High-level)
- Can the user understand how to organize bulk files efficiently?
- How quickly and easily can users add patient’s files to a specific app and add new apps to the home screen of the PHIgo app?
- How easily and successful are the participants to manipulate the patient’s lung images and perceive the feedback given by the app?
Participants & Test Design
Due to the PHIgo non-disclosure agreement, all participants were provided by our client. According to the product attributes, we chose the " 1 Product – 1 Group – 1 Set of Tasks" methodology to design the tests. All the participants operated the PHIgo Version1.1 and performed the same set of 4 tasks
Tasks & Training
We broke down the 3 high-level research questions and integrated them into 4 tasks.
1
Find a medical record under a patient’s name and delete it.
2
Measure the size of tumors
3
Delete all annotations on the patient’s lung image
4
Add the file to the “NovoStroke” app from the “Application Center”
Since the PHIgo is a highly specialized medical software and all participants were first-time users, we designed a 3-step training before the test to help them get a basic understanding of how PHIgo works.
- Import files from computer and add them to external storage
- Add 2 folders to local storage
- Add a file to an app that is already in “Recent Used”
Test Environment & Roles
Due to Covid-19 restrictions, usability testings were conducted online via Zoom. Since the PHIgo application only ran on the company computer, a GE technical supervisor helped a lot during the tests.
- The technical supervisor was responsible for setting up the device and helping the participant with any related technical support
- The moderator was responsible for guiding the participant throughout the tests
- The trainer was responsible for training participants before the official tests
- The observer was responsible for taking notes
Test Materials
(1) Background Questionnaire: collecting participants' demographic information
(2) Pre-Test Questionnaire: understanding how they are familiar with medical image processing software and photo editing software
(3) Post-Task Questionnaire: asking for the difficulties, expectations, and feedback for each task
(4) Post-Test Questionnaire: letting participants rate the overall experience and their satisfaction with PHIgo
Execute & Refine
After conducting the pilot test, we found some problems. For example, if the participant stuck with a task and we didn't give any hint, it would take too long to determine the success/failure of the task. To improve the test efficiency, we decided to give the participants hints if they did stuck in tasks such as they didn't notice something on the screen.
Other refinements on our test plan and test materials were:
Background Questionnaire:
Task 2 Scenario
Post-Task Questionnaire
Findings
For the quantitative data that we collected from the surveys, we used several methods to analyze and present the results, such as System Usability Scale Analysis. We did across group analysis on the overall experience. Qualitative data such as verbal feedback and observation also contributed to the findings.
System Usability Scale Analysis
SUS consists of a 10 item questionnaire with five response options for respondents; from Strongly agree to Strongly disagree. Based on the responses from all participants, we have this result:
Recommendations
Usability issue list from the most critical to the least critical
Based on the heuristic evaluation and the findings from the 5 usability tests, we have the following recommendations:
- Place progress bar in the central vision of the user
- Place the “application center” icon near (above) the Recent Used App
- Add dividers between different columns on the file list
- Simplify the delete process by right-clicking or using the keyboard shortcuts
- Use user’s language: import, WS Data Manager, Scan Dicom
- Change cursor to drawing marks instead of arrows when drawing annotations
CONCLUSION
What I did
It was great to work with my two groupmates on a real brand-new product. What I did in the group work were:
- Proposed research questions
- Conducted heuristic evaluation of PHIgo homepage
- Designed the 4 tasks (scenario, success criteria, and data metrics)
- Moderated 2 usability testings
- Trained participants in all test sessions
- Designed post-task & post-test questionnaires
- Analyzed collected data of task 3 & task 4, and summarized the insights
In addition to the above work, I also took the role of a facilitator for both clients meetings and group meetings. The role made me understand what product managers do. I set a timeline for the project, communicated a lot with stakeholders, recorded discussion notes for all meetings, and coordinated assignments for each group member. The most rewarding thing was the client iterated PHIgo based on our heuristic evaluation report in only two weeks. Clients appreciated our professional test and design recommendations which did help to improve the user experience of the product.
Next Steps
Formative usability tests in next versions
- The agile design team has made 27 modifications based on our heuristic evaluation report.
- Do formative usability tests in the early/middle stage of PHIgo 1.2
A/B tests
- Avoid unwanted risks by enabling clients to target their resources for maximum results and efficiency
Participants from a wider aspect
- Participants are supposed to be composed of researchers and doctors with a medical background
- Doctors might not familiar with the built-in apps in PHIgo
More participants
- To cover more usability issues and collect more suggestions/expectations