London Web Design & SEO Services

What happens in a typical usability test session?

Here is what happens in a typical one-hour usability test session:

• The facilitator:

• welcomes the participant and introduces anyone else who is in the room
• invites the participant to sit in front of the computer where the participant will be working
• explains the general goal of the session—to have the participant try out a Web site (or whatever the product is that is being tested)
• asks participant profile questions and has the participant sign the release form
• explains thinking aloud (and may demonstrate it and have the participant do a think aloud exercise)
• asks if the participant has any questions before starting and answers any that will not give away what you want to learn from the participant
• tells the participant where to start

• The participant starts to work with the Web site (or other product). The facilitator may ask some initial questions or give the first scenario (whatever you have planned).
• The participant works on the scenario while thinking aloud. The note-takers take notes.
• The session continues from scenario to scenario until the participant has done (or tried) them all or the time allotted has elapsed.
• The facilitator asks the end-of-session questions and then thanks the participant, giving the participant the agreed-on incentive, and escorts the participant out.

What makes a good test facilitation?

Review the following section to learn more about how to properly facilitate a usability test.

Treating participants with care

The most important thing to remember is to take good care of participants. Make them comfortable. Treat them with respect.

They are helping you by trying out the prototype of your product. They may struggle. Indeed, the reality of usability testing is that you watch a few people struggle so that the many, many people who will use the site later will not have to struggle.

Staying neutral

You must also remember to remain neutral. You are there to listen and watch, not to demonstrate the site; not to train or teach; not to say either negative or positive things about the site. Your only talk should be to encourage the participant to think aloud. If the participant asks a question, reply with something like “What do you think?” or “I’m interested in what you would do.”

Deciding when and how much to help

You also cannot jump in and help participants immediately. Watching participants try to figure out how to do tasks is what usability testing is all about.

If the participant gives up and asks for help, you must decide whether to end the scenario, give a hint, or give more substantial help. The key is to make the participant comfortable while not giving away information that might change what will happen in later scenarios. Remember you will not be at the desk or in the home of the users who get the product outside of the usability test. A good solution is often to say, “Thank you for what you have done. We have learned a lot from that. Let’s go on to a different task.”

One of the decisions that you and the team should make in planning is how much of a hint you will give in each situation and how long you will allow participants to work on a scenario when they are clearly going down an unproductive path. If you stop participants too soon, you will have learned little about their way of thinking about the site. If you let them go on too long, they may not get to other scenarios that you also want to learn about.

Taking good notes

Note-takers should capture what the participant did in as much specific detail as possible.

If you have put the optimal path and alternative pathways into the note-taking form, note-takers can just check off what participants do when the participants act as expected. When participants do not take one of the expected pathways, however, it is most useful to note exactly what they do.

Useful note: Clicked on link to Research instead of Clinical Trials.
Not useful note: Clicked on wrong link.
Not useful note: Was confused about links

The explicit note gives us information about the participants’ view of the site.

Note-takers should also capture insightful comments that participants make. Note-takers should try to capture what participants say in the participants’ words. One of the benefits you get from watching and listening to users is to hear the words they use for what they are trying to do.

Even if you are tape recording the sessions, it would be very time consuming to go back to the tape to do a detailed analysis after the sessions. The more and better notes you take during the sessions, the easier analysis will be.

It is important to note that some usability testing tools will log some of this data for you, including users’ paths, time on task, number of pages viewed, task completion rate, satisfaction scenes, and more.

Next steps

When you have conducted the usability test sessions, you will be ready to Analyze the Results.

Make sure you have everything you need

Use the test plan to create a list of everything that you need to do to set up for the test sessions.

Usually, you must be sure you have:

• the prototype you are going to test
• the computer set up for the participant with the monitor, resolution, and connection speed that you indicated in the test plan
• note-taking forms on paper or set up on a computer
• consent forms for participants to sign and a pen in case the participant does not bring one
• questionnaires, if you are using any
• the participant’s copy of the scenarios
• cameras, microphones, or other recording equipment if you are using any
• folders to keep each person’s paperwork in if you are using paper

Do a dry-run and a pilot test

If you are at all worried about the technology, do a dry-run without a participant. A dry-run allows you to test the technology and to get a good sense of the scenarios and the Web site.

Then do a pilot test. In a pilot test, you learn:

• whether your questions and scenarios make sense to the participant
• how long participants are likely to take for each scenario
• if the technology is working as you expect it to

Also, in a pilot test, the facilitator and note-taker(s) get to practice their roles. Do the pilot test with a participant from your less experienced user group because that way, you are more likely to see any problems with your materials or your timing.

Do the pilot test at least a few days before the main data collection is scheduled to start so that you have time to change the scenarios or other materials if necessary.

If the pilot test is different from the regular test, do not count the pilot test participant’s data, especially not in your quantitative results. If you do not make changes after the pilot test and the pilot test was very much like the other test sessions, you may want to count the pilot test participant’s data.

Next steps

Once you have set up, you are ready to Conduct the Usability Test.

Who should participate in a usability test?

In a usability test, the participants must be like the people who will use your site. The point of usability testing is to find out what is most likely to happen when your site gets out there to be used.

If the participants in your usability test are not like the people who will use your site, you will not find out what you need to know. It is quite easy to get false results. The most common is the false positive you are likely to get if you have internal staff testing a site meant for an external audience. (That is, the site works well for these participants in the usability test but fails with the real users after it is launched.)

Who should recruit participants?

Getting the right people to take part in a usability test takes time and effort.

If the team (management, development, usability) has access to representative users, that’s a fine way to recruit. Just be sure the people you get that way really represent the people you are concerned about being successful at the site. And use a screening questionnaire so everyone who is recruiting is asking the same questions of potential participants.

If the team does not have access to representative users, you may want to have a commercial recruiting company find the participants for you. Most recruiting companies require two to three weeks to find the necessary number and types of participants. And you may be asked to provide the recruiters with a screening questionnaire.

What should you ask in a screening questionnaire?

For templates you can use, see Usability Test Screeners. The questions are examples, taken from government Web site screeners, that might be included in your screening questionnaire; you may want other questions or a different mix of participants for a usability test of your site.

Does recruiting cost money?

Usually when having others recruit for you, plan on a cost associated with finding the people, another cost for each participant’s time (incentive), and in some cases for travel/parking expenses.

If you do not use a commercial recruiting firm, you may still need to plan on incentives to get participants to come. This may be money and/or travel expenses. It may be a non-monetary gift.

The right incentive is what will entice the people whom you want to participate to say “Yes, I will come try out the Web site.”

Next steps

Now that you have recruited the right participants, you should also Create Final Scenarios. If you have both participants and scenarios, you are ready to Set Up for the Test Sessions.

What is a scenario?

In usability testing, participants use the Web site (or prototype or Web application) to do tasks. The scenarios are the means for telling participants what tasks to do. During the test session, you give the participants the scenarios—either in writing or verbally, one at a time, as the participant is ready to do that scenario.

What makes a good scenario for usability testing?

A good scenario for usability testing gives the participants:

• a goal/task (what to do or what question to find the answer for)
• data, if needed, that a real user would have when going to the site to do that task

You can give the scenario as just the statement of the goal/ task or you can elaborate it a little with a very short story that adds motivation to get to the goal.

Here is an example of a scenario in both forms: brief (no story/motivation) and elaborated (with story/motivation). It comes from a usability test of GSA.gov.

Brief (no story/motivation):
In which government building can you find Bertrand Adams’ 1937 painting Early Settlers of Dubuque?

Elaborated (with story/motivation):
Your grandfather told you that he posed for Bertrand Adams when he was painting his large 1937 masterpiece, Early Settlers of Dubuque. You heard that the painting is displayed in a Federal building. In which building can this artwork be found?

Both types (brief and elaborated) work in usability testing. Both seem to elicit desirable behavior from test participants.

Participants enjoy the story-type scenarios, but keep them short. You do not want reading the scenarios to take up much session time. You also want to be sure that all participants understand the scenarios.

What does NOT go in a scenario for usability testing?

The scenarios do not include any information on how to accomplish the task. That is what usability testing is for—to show you how the participant goes about accomplishing the task.

Should you write down how to accomplish the task?

Yes. In the material that observers and note-takers use (but that the participant does not see), it is a good idea to include the pathway (or alternative pathways) to accomplish the task. This helps observers know what to expect and saves considerable time for note-takers.

For example, for the scenario about where to see the Adams painting in Dubuque, we might write down the following pathway where each step assumes a click that moves the participant to a new Web page:

1. Home page top tab: “Citizens”
2. Right panel: Art
3. Left panel: Fine Arts Collection
4. Left panel: Inventory Online Database
5. Center: Visit Now
6. Find: Bertrand Adams

What if you are giving participants choices for answers?

In most usability tests, participants let you know they have successfully done the task by telling you the answer that they found. In some cases, you may want to give participants multiple-choice questions.

If the scenarios are formatted to require participants to answer multiple-choice questions, the possible answers must be given to the participant at the end of the scenario. In the participant’s copy, of course, the correct answer is not designated. In the test plan, these questions should be included with the correct answer designated.

For example:

A) Customs Office
B) Federal Correctional Center
C) FBI Office
D) Post Office & Courthouse

What if users have difficulty understanding a scenario?

A good scenario is one that participants understand even if they have difficulty completing the task.

To make sure that your scenarios are clear, try them out in a pilot test. (We discuss pilot testing in the article on how to Conduct the Usability Test.) If you find that participants do not understand a scenario, rewrite it. If you find that many of your scenarios are not clear to participants and that you must rewrite several, do a second pilot test.

Be sure to distinguish between a scenario that is not clear and a task that is difficult. Unclear scenarios do not tell you anything about the usability of the site. Difficult tasks do. Rewrite unclear scenarios. Leave difficult tasks in the usability test if they are going to help you understand how the site needs to be improved.

After running the first test with these scenarios, it is usually a good idea to go back and put a copy of the final set of scenarios (those actually used in the test) into the test plan.

Next steps

Now that you have final scenarios, you should also Recruit Participants. If you have recruited participants, you may be ready to Set Up for the Test Sessions.

The first step in each round of usability testing is to develop a plan for the test.

Don’t make it a long narrative that is difficult to read. Use headings and bullets so it is easy to scan.

The purpose of the plan is to think through and write down what you are going to do and get approval for that so management and the entire team (design and usability) all agree.

If the test plan is well-prepared, much of it can be reused as the background information in the report after the test.

Who prepares the test plan?

Typically, the usability specialist meets with the rest of the team to decide on the major elements of the plan. Often, the usability specialist then drafts the plan, which circulates to management and the rest of the team. Once everyone has commented and a final plan is negotiated, the usability specialist revises the written plan to reflect the final decisions.

What goes into a plan?

This table gives you a quick overview of the parts of a typical plan. The rest of this article gives you more detail about each of these parts.

The reason for including this information in your plan is for management and other team members to understand and comment on what you plan to do. If they raise questions or issues about the plan, you must negotiate to resolve any questions or issues. In the end, your plan will drive the usability test and you do not want management or other team members to have expectations of the test that you will not be meeting.

Scope

Indicate what you are testing: Give the name of the Web site, Web application, or other product. Also specify how much of the product the test will cover. Examples might be:

• the prototype as of a specific date
• just the navigation
• navigation and content

Purpose

Identify the concerns, questions, and goals for this test. These can be quite broad; for example, “Can users navigate to important information from the prototype’s home page?” They can be quite specific; for example, “Will users easily find the search box in its present location?”

In each round of testing, you will probably have several general and several specific concerns to focus on. Your concerns should drive the scenarios you choose for the usability test. (Of course, when conducting the test, also be alert to any usability issues that emerge that you had not included in your original list of concerns.)

Schedule and location

Indicate when you will do the test. You may want to be specific about the schedule. For example, you may want to indicate how many sessions you will hold in a day and exactly what times the sessions will be. (You will need that information to recruit participants, so you must decide the schedule early on.)

Also indicate where the test will be held. Again, you need that to recruit participants as well as to let observers know where to come to see and hear the test.

How long should each session be? Typically for an informational Web site, participants come for one hour. To test a Web application, you may want to schedule 90-minute sessions. Leave yourself a little time between sessions to reset the environment, to briefly review the session with observers, and to take care of personal needs.

How many sessions should you do in a day? Watching and listening to people working with a product can be exhausting, particularly if you are watching participants really struggle. Some teams schedule as many as 15-16 sessions in a single day. Others limit themselves to two to four sessions in a day.

Do we need a usability lab for testing? No. You do not need a lab for testing. If you have access to one, use it. Most usability labs have cameras, microphones, recording equipment, etc. so they have an environment all ready to go. Most also have separate observation rooms where people other than the participant and facilitator can watch the test. These days, rather than having one-way glass to watch through, many usability labs feed the audio and visual into the observation room. If you do not have access to a usability lab, you can still do usability testing. You can test in a conference room, hotel room, the participant’s work or home space, or other venue where you can arrange to meet users.

Participants

One of the most important parts of the test plan is a clear indication of the number and types of participants to be tested. This can range from a general set of user profiles to specific personas that represent each of the major user segments. The more precise the user descriptions, the easier it will be to develop a screening questionnaires that will bring you the types of people you need as test participants.

How many participants? The number of participants is frequently dictated by the budget and time allotted for the test. If you can have as many participants and as much time as is needed, you can calculate how many that should be. We give you three useful methods here. Even if you will be constrained by budget and time, you may want to understand these three methods so you can decide which is appropriate for the test you are doing.

Method 1 relates the number of participants you need to the number of usability problems that you expect to uncover. According to the formula behind this method, if you are testing an early prototype or a particularly difficult site, you need only a few users (four to six) to uncover the most serious problems.

Method 2 is the one to use if you must make strong inferences from the participants to a larger population. For example, if it is important to feel confident that certain usability objectives are truly met, then using “95% confidence intervals” is a good choice. This could show that a minimum of 12 to 15 people will be needed for each unique group (type of user) and possibly as many as 20 to 30, or more.

Method 3 is the one to use if you must ensure that a usability test has sufficient statistical power to identify real differences when they exist. This method is most appropriate in comparison testing, when you are measuring against a previous product or against competitors. In this case, the number of people could be at least 35 (or more) in each unique group.

Scenarios

Scenarios are very short stories that motivate participants to attempt tasks with the product. At this stage (drafting the test plan), concentrate on listing the tasks that you will test through the scenarios rather than on the final wording of the scenarios. We cover how to Create Final Scenarios in another article.

How do you decide on tasks for a usability test? Consider these three questions:

• How does the task relate to one or more of the items that you listed in the purpose section of the plan?
• How frequent and/or how important is each task for users of the Web site?
• How susceptible a particular Web site element to a usability issue? For example, you may want to test items that you think are a usability issues to see if they really are.

Relate each task to the items in the purpose section. The purpose section tells you what you are trying to learn about the Web site. The tasks and scenarios are the vehicle for that learning. A task (scenario) is only valuable for the usability test if it is going to help you answer one or more of the concerns, questions, or goals you listed in the purpose section. (One scenario may address more than one of the items under purpose. You may have several scenarios that address one of the items. The relationship is not 1:1, but you must have at least one scenario for each item in your purpose section and every scenario must address at least one of your concerns, questions, or goals for the test.)

Select scenarios that rate high on frequency and/or importance. The Web site must allow users to do their most frequent and most important tasks, so you should concentrate on those in testing the site. You can find out which are the high frequency and most important tasks through your Task Analysis (for example, user surveys and contextual interviews and observations of users at work) as well as other usability activities, such as Card Sorting. You may also get useful information from Web logs, search logs, and other ways of tracking what users do at your site.

How many scenarios do you need? For a typical one-hour usability test of an informational Web site, you probably want to end up with about ten scenarios. (Of course, depending on your Web site, users may be able to accomplish more scenarios - or fewer - in the time of your testing. Consider how long a task will take typical users to complete.) Consider conducting a pilot test to see how many scenarios can be realistically in one hour.

How many scenarios should we include in the draft plan? For your first draft of the test plan, consider including several more than you will use (perhaps as many as twice what you need). This will allow you to discuss the relative importance and value of different tasks/scenarios with management and the rest of the team. Through negotiations (dropping, adding, changing tasks/scenarios), you should arrive at a set that everyone agrees are:

• realistic
• relevant to your purposes
• frequent and/or important to users

Lastly, the test plan should indicate whether the scenarios will be presented in random or sequential order.

Questions

The plan should include the types of questionnaires you are going to include, when and how you will ask the questions, and the specific questions you will ask. If you are going to use pre- or post-test questionnaires, you should include that in the test plan.

Most usability tests include three types of questionnaires:

• Participant profile questions
• Questions when the user first sees the site
• Questions at the end of the session

Participant profile questions. These allow you to validate that the person represented himself or herself correctly when recruited. You may also want to add a few other questions to learn more about the participant than you covered in the recruiting screener. Participant profile questions might be about age, gender, education, work experience, and computer or Web experience.

You can ask participant profile questions as:

• a questionnaire that the participant fills out on arriving for the session
• in a brief interview at the beginning of the session

Advantages of using a written questionnaire:

• can be done before the session starts (saving session time)
• no interviewer bias

Advantages of asking the questions as an interview:

• serves as an “ice-breaker” to make the participant comfortable
• can probe to be sure information is accurate and useful

Questions when the user first sees the site. You may want to get the participant’s initial impressions of the site by asking a few questions before starting the scenarios. These might be:

• What site is this? How can you tell?
• Who do you think the site is for? How can you tell?
• What do you think you will find at this site? How can you tell?
• What are your initial impressions of the site?

Questions at the end of the session: You probably want to get participants’ impressions of the site after they have worked with it. You can do this with:

• open-ended questions
• closed questions using radio buttons, check boxes, or a Likert scale (rating something on a numerical scale)
• a standardized satisfaction questionnaire

If you ask open-ended questions, it is critical to ask them neutrally, so you do not bias the participant to give you a particular answer. Here is a set of six open-ended questions for a usability test that focused on navigation and search:

• What is your overall impression of the site?
• How would you describe finding what you were looking for on this site today?
• What is your impression of the search capability?
• What did you like best about the Web site?
• What did you like least about the Web site?
• If you were the Web site developer, what would be the first thing you would do to improve the Web site?

Data to be collected

In addition to qualitative notes about what you see and hear in a usability test, you probably want to collect quantitative data.

Success measures. Indicate in the report how you will determine “success” for each scenario.

Time measures. Indicate if you are going to time each scenario and what time measures you are going to take. (Total time to complete the scenario? Separate time figures for navigating and for understanding [time spent on content pages]? Time to recover from an error?)

Error measures. Indicate if you are going to measure errors and what you are going to count as an error.

Number Pages. Indicate if you are going to count the number of clicks to information.

Pathway. Indicate if you are going to need users’ paths through the Web site.

Satisfaction Rating. Indicate if you are planning to administer a satisfaction questionair.

Set up

The test plan should include information about the computer system you will be using. This should reflect the typical system that most users of your site work with in their daily lives. Include information about:

• monitor size and resolution
• operating system
• browser

You should also indicate if you are planning on videotaping or audiotaping the test sessions and if you will need any special usability testing tools.

Roles

Include a list of the staff who will participate in the usability testing and what role each will have. The usability specialist should be the facilitator of the sessions. The usability team may also provide the primary note-taker. Other team members should be expected to participate as observers and, perhaps, as note-takers.

Next steps

Once you have a test plan in place, you are ready to Create Final Scenarios and Recruit Participants.