Navigation Advanced Common Sense home page Advanced Common Sense home page Consulting services
Latest Topics
 
 
 


Reply
  Author   Comment  
Sandy

Registered:
Posts: 6
Reply with quote  #1 
Many of my participants get a little freaked out by a video testing session.  Then if I tell them they are being observed they squirm a little more. 

So, my question is... are we obligated to tell our testers they are being observed during a video testing session? Do you think that the fact they know they are being watched skew the results.  Will not telling them hurt the credibility of your efforts? In other words, if you don't tell them they are being observed and then they find out later (inevitably they will with my in house program), will they trust you to test with you again?

Steve Krug

Avatar / Picture

Proprietor/Landlord
Registered:
Posts: 34
Reply with quote  #2 

Good question. The last thing we want is participants getting freaked out.

First, let's get one thing out of the way: if they're being recorded or observed by other people, you *have to* let them know. No question about it. It's simply part of your ethical responsibility to your participants. And they have a perfect right to choose not to participate when you tell them. So it's not a question of whether they might find out later. Again, you *have to* tell them.

Then when you say "inevitably they will [find out later] with my in-house program," it raises two other questions for me:

  • Are you bringing the same participants back to test in later sessions (or "reusing them" as I usually call it)? In most circumstances, you won't want to use the same participants again—at least not to test the same product.

  • Are you using employees of your own company as test participants? If you are, you have an even stricter ethical responsibility. For instance, you have to make sure that they won't be made to look bad in front of their fellow employees, or even worse, their bosses.

The other question is what do you mean by "a video testing session"? Do you mean that you're using a WebCam to record the participants' faces during the test in addition to what's going on on the screen? I suspect that's what you mean, since it's extremely rare for people to get freaked out just because the screen and their voice are being recorded. In fact I'm always surprised at how casual people tend to be about it.

On the other hand, recording their faces *can* make some people (although usually not that many) uncomfortable. And personally, I don't think it's a good thing to do in general. I suspect that having the image of the participant in the corner of the screen (picture-in-picture, or PIP) actually tends to be a distraction for people who are observing the test or watching recordings after the fact. It's hard to take our eyes off people's faces, but your observers really should be focusing on what's going on on the screen and what the participant is saying. (But this—whether or not to record faces—is a much larger topic, so I'm not going to say more than that here.)

Hope that helps.

Sandy

Registered:
Posts: 6
Reply with quote  #3 

Thanks Steve.  Yes this helps.

 

Just to clarify, we were doing PIP but I've decided to discontinue as it is more of a distraction and you are correct, more of a freak out factor than just voice and screen capture alone.  Besides, I haven't found physical reactions to be that valuable in my findings.

 

My situation is I work for a government contractor, and we are testing intranets and internal web applications, so all testers are in house and although I try not to use the same person more than once, there are times when this is necessary.  What I meant by them inevitably finding out is I'm evangelizing the user testing process, and it's a topic of discussion more and more around the water cooler, etc... so anyone who has been through the testing process will most likely talk to other employees about it. 

 

I have been letting them know they are being observed. In most cases, the tester is an engineer and we are baseline testing a current app before we do a redesign. These application are very robust. As you mentioned, the tester is often concerned that his boss or the project "customer" who may be a coworker is watching him during testing of an application that he/she is supposed to be familiar with. They don't want anyone to know they may not know everything about the application.   Although you and I may see this as a failure of the app design, the users may be concerned that this will be perceived as lack of expertise.  I think perhaps the key here is for me to put the user at ease by explaining that they are not being tested, the app is and, also advise the observers of the same. The focus has to be on the app, not the tester.

 

I do have them sign a video recording release but had not included the observation part of the test in the "i'm ok with this" release document. I'll start doing that as well.

 

Thanks again Steve.  Sure hope more people drop in on this forum. It could be my life line ;-) 

Steve Krug

Avatar / Picture

Proprietor/Landlord
Registered:
Posts: 34
Reply with quote  #4 

Quote:
Originally Posted by Sandy
Besides, I haven't found physical reactions to be that valuable in my findings.
 

Glad to hear you say it. I think tone-of-voice is a much better indicator of state of mind, which is why you wan to always have the clearest audio you can get.

Quote:
Originally Posted by Sandy

As you mentioned, the tester is often concerned that his boss or the project "customer" who may be a coworker is watching him during testing of an application that he/she is supposed to be familiar with. They don't want anyone to know they may not know everything about the application.   Although you and I may see this as a failure of the app design, the users may be concerned that this will be perceived as lack of expertise.  I think perhaps the key here is for me to put the user at ease by explaining that they are not being tested, the app is and, also advise the observers of the same. The focus has to be on the app, not the tester.
 
Exactly right. And in an internal situation like that, you also have to exercise a lot more control over videos or clips floating around for the same reason (i.e., their boss might see it). You never want to identify the participant by name in the videos. And not having the particpant's face in the video makes it much easier to protect their anonymity.

Quote:
Originally Posted by Sandy

I do have them sign a video recording release but had not included the observation part of the test in the "i'm ok with this" release document. I'll start doing that as well.
 

I'm not sure you really neeed to include the "observed" part in the release. But it's crucial that you mention it in your introduction. Are you using the script from Rocket Surgery? (You can download it from my site.)

Quote:
Originally Posted by Sandy

Thanks again Steve.  Sure hope more people drop in on this forum. It could be my life line ;-) 
 
Congratulations on being the first one here! Thanks for contributing.

MikeHogan

Avatar / Picture

Registered:
Posts: 6
Reply with quote  #5 

I just want to mention that we have all participants sign a release form before we conduct the test. I would also recommend having someone vet it from a legal perspective rather than just making one up on your own.

Steve Krug

Avatar / Picture

Proprietor/Landlord
Registered:
Posts: 34
Reply with quote  #6 
Quote:
Originally Posted by MikeHogan

...we have all participants sign a release form before we conduct the test. I would also recommend having someone vet it from a legal perspective rather than just making one up on your own.

Funny. I usually tell people to try to stay *away* from legal if at all possible (although I'm usually talking about NDA's, which often scare participants off).

You don't have to make one up on your own, though. There are a lot of them out there, and most cover the same few points. Mine (http://sensible.com/downloads/permission-form.doc) is probably the shortest, because it only covers them giving permission to be recorded. Dana Chisnell and Jeff Rubin (http://media.wiley.com/product_ancillary/81/04701854/DOWNLOAD/9780470185483c08.recording_permission.zip) go into a lot more detail.

Some others actually explain the participant's rights (leave at any time, for instance), but I'd rather cover those things in the opening part of the script you read to them. Making them read and sign a complex form feels to me like getting things off on the wrong foot.

MikeHogan

Avatar / Picture

Registered:
Posts: 6
Reply with quote  #7 

I like the example you site from Sensible. It's pretty close to the one we use as well. Anything more than that gets a little intimidating I would think and might scare off some folks.

Previous Topic | Next Topic
Print
Reply

Quick Navigation: