User Testing Case Study

Problem & Solution

In this case, there is a communication gap. And my role in it should be the communication bridge. So this time, I will try to cooperate with the vendor design team and use their prototypes, which the userflows of some cases are more controversial, to do a user testing with end users, and get some qualitative results and insights through this test.

Goals

1. Get to know how end users use the app
2. Get to know the feedbacks from them about each version of each flow

My Role

Researcher (me) | Moderator | Observer & Coordinator

• Planning and execute the testing
• Briefing, explaining and guiding participants during the research process
• Setting up the venue, observing participants, recording video and taking notes
• Deliver research insights/findings through research report

Since this prototype is made by the vendor design team, they want to find a better userflow, so there are 3 flows to be tested, and each flow has 2 versions. See which version is better, or what could be improved.

At the meeting, in addition to giving me a demo of the prototype, they also told me about

• The results they wanted - Pick a better version or give an insight / testing report or give some actionable comments on the design;
• The questions they want to know, such as they want to know if the button on a certain page is noticed by the user
• The recommended device

When asked what form should be used for testing with users, the most ideal way, I think, is to use the same device for each user, and then I guide them to complete some tasks next to them, which can achieve the best results. But the problem of PM concern is that I don't know if the end user has the time or the venue to do such a thing, if it is possible to use the teams sharing screen. On this point, I have reached a consensus with the design team that in-person moderated research will be more effective and efficient. Therefore, I also ask PM to help contact the user, let me introduce our methodology to the stakeholder, and then make an appointment for testing (Thank you very much 🙏) Afterwards, they also sent us a User Testing Report, introducing the script of each flow and the tasks that can be done. I also played with prototype to get familiar with it.

Questions

Prepare for the questions I might ask in advance. I remembered that, it seems that these questions were not asked in order 😂, but I also covered them in the process, which is great. And during the testing session, I found that other questions were also asked, which were not expected. It was based on different scenarios and situcations that I could learn more.

Data Contents

My train of thought as I prepared this part was, what am I going to write down. Including User Feedbacks, Observations, and User Profiles. User Profile includes the type of user (whether it is an end user or not), whether there is any experience in using the old version of the app, what device do you usually use, the frequency of contact with this matter, the age group, and the location where you work, etc.

Pilot Test

I happen to know someone who does the same job! Thank god! I asked him to do a Pilot Test that night. After completing this Pilot Test, I will also jot down the data. And when recording data, I adjusted the structure of Data Content and added "How". I want to write down the specific steps completed by the interviewee, because this can be used as a test report for the vendor design team to refer to.

Through this Pilot Test, in addition to practicing the way of asking questions, you can also get some insights in advance. After the subsequent test is completed, it is found that the insights of the Pilot Test may be effective (it's amazing). Therefore, you already have a certain understanding about the user workflow. And there are some preliminary recommendations occurred, although I don't know if I will use it later, it is always good to write it down.

Introduction

• Say hello, introduce yourself
• Introduce what the test to do is about, there are 3 cases in total, and each case has 2 versions

Do the task

• let the participants complete the corresponding tasks for the cases

Follow up questions

• Know about their user profile
• Any questions from both sides
• Chit Chat (there may be feedback, for example, a participant accidentally revealed some similar apps he has used before, which can be used as a reference)

• After each participant has completed the test, I will ask U to ask her any questions she wants to know. And I've found that every time we're done with a participant, we have some deeper questions that come up. For example, after I finished the demo with U, we would find that some complicated cases appeared, then every participant afterwards, we would ask them how to deal with these complicated cases.
• Then I'll focus on what the previous participant did wrong and see if the same thing happens to the next participant.

• After each participant has completed a set of tests, they will have their own behavior, and I will ask them why they do it based on these special behaviors. For example, an participant clicked “Edit” button after clicking the checkbox, I asked him why he clicked the “Edit” button, he said "just want to know what's in it" 😂 interesting
• After each participant finishes a case, I firstly noted which version they prefer and why, as well as their specific quote and feedback

 

• An interesting pattern was also found
  • The two participants who had not used the old app chose v2;
  • And those who had used the old app chose v1

The reason is that some designs of v1 are the same as the old app, users get used of the old app's behavior

2.1

It is found that in case 1, each participant will have the same pattern/action. When you want to mark a defect, first press the location and then select the type. Therefore, the proposed flow should take this into account

2.2

Most respondents will press the "save" button after completing an action, but it's a button they probably don't want to press because it's an extra step. Therefore, I will propose auto save or replace the content of the button and set it in the last step, which means that the button is pressed after they have completed all actions on this page.

2.3

Several participants often want to press the collapse button to collapse the panel. It is conceivable that users prefer simple pages to view the floor plan.

3.1 - In version 1 of case 2, some participants may not know that the checkbox can be clicked, or that the checkbox is used for multiple selection and then do the next action. Therefore, I will propose to close the checkbox with an action button, for example: select to mark, the checkbox will not appear until the button is pressed

3.2 - There are two very long buttons in version1 of case2. After testing, it is found that the participants hardly care what the content of the button is, but when they press the checkbox and do the next action, they will read the button to confirm whether to do this action. Therefore, I will make the wording of the button of the proposal page concise and to the point, do not let the user spend time reading and thinking about the text content

4.1

Version 2 of case 3 is to switch from horizontal view to vertical view to select the next item. Many participants thought that it would be a little uncomfortable to turn the screen of the mobile phone around. Therefore, I will propose a horizontal view page for the user to select the next item

4.2

The version1 of case 3 is when the user presses a submit button and jumps directly to next project. Many participants believe that this will restrict them from going to the next project, and they do not want to go to the next project in order. They prefer to choose which project to go to. Therefore, the horizontal one proposed in #4.1 The page just fits

4.3

The version1 of case 3 is when the user presses a submit button and jumps directly to next project. Many participants believe that this will restrict them from going to the next project, and they do not want to go to the next project in order. They prefer to choose which project to go to. Therefore, the horizontal one proposed in #4.1 The page just fits

1. Fortunately, I did the Pilot Test.

The Pilot Test is really important, it allows me to practice once, to get familiar with how all the cases work, what the story is like, what questions I should ask, etc. It also allows me to foresee possible situations and problems that participants may encounter.

2. Practice is really important.

This time, I found myself learning how to ask questions and guide the participants to do actions, which can be neutral.

• It still needs to be practiced more, because sometimes there is a slight bias, intentionally or unintentionally. For example, when the participant presses a place that does not respond, he may panic, and then press other places. Sometimes he wonders if he is doing something wrong. At this time, I will guide him to take another action, or use another way of thinking, I would say: "When you can't click here first, then you need to find XX first, then what will you do?" When he received the other way I said, he would instinctively ask me "Is it like this / Is it here / Oh it's here" for asking the confirmation of what he found, and then I would also instinctively answered "That's right!/That's it!" 🙊 which I shouldn't be acting like this 😂

3. With Stakeholders doing the test together, it would be better if they play a role as observer.

Because it can let them know how users use and think, and listen to more opinions other than them. We don't need to persuade them to use such design, instead, they will understand from the testing

4. After each interview with a participant, you should write down the feedback roughly, summarize where there are problems, and observe whether these problems will appear when the next participant is doing the test.

5. It would be better to have one more person to help. In fact, it would be really difficult for one person to execute the test, because if you want to be a moderator, you also need to be an observer, and you have to take notes.

6. Be sure to choose a quiet place and set up your device.

7. Be brave to ask questions, ask if you don't understand, ask if you don't understand, maybe they'll be happy to tell you