A long time ago, I was offered a job in a qualitative research company. I ended up not taking it, and returned to academics. That part of it is another story, but the company hired me through campus recruitment. They wanted someone with my set of skills. I remember being happy that they understood the value of well-done research with a solid foundation.
Flash forward about ten years or so. To last week, in fact. I was wandering through a mall with a friend; let’s call her Amy. A woman holding a clipboard came up to us and asked us, “Do you want to do a survey? You’ll get seven dollars! It’s about product packaging, and it will only take twelve minutes.”
I confess I have a hard time refusing people who administer surveys. I’ve done their job, when I was in college. I spend days calling people and trying to get them to answer surveys. It’s a terrible job. I had to meet a quota, they didn’t want to talk to me, many people yelled at me, and (as is generally the case with surveys) I didn’t have enough responses. It never occurred to me to fudge my data, but I discovered later that some other people who were doing the same job filled out some of their surveys themselves. It was a few more years before I really came to realize that that rendered the data meaningless.
Anyway, so here are Amy and I in the mall, and we say, “okay, we’ll take your survey, we could use seven dollars.” So she turns to me and this is how the conversation goes.
Woman: How old are you?
Me: XX years old (you don’t need to know, gentle reader)
Woman: Do you earn more than $XX000 in the year as a household?
Woman: Have you ever used X brand of paper plates?
Me: (thinking very hard now) I think so.
Woman: Do you use them more than five times a week?
Me: (horrified at the thought of such waste) No!
Woman: Have you ever used body wash?
Me: (okay, here’s what we’re really about) Yes.
Woman: Do you use it more than five times a week?
Woman: (to Amy) How old are you?
So at this point, I’m a little stumped. I’ve realized a few things:
1. She’s interested in body wash. I don’t use body wash, but it’s not like I don’t shower more than five times a week. I simply use bar soap. I already see an area in which their survey fails, which is that it could be a vehicle for increasing the product market, i.e. bringing the body wash to the notice of someone who hitherto does not use body wash on a regular basis. Would the new packaging do this? Aren’t companies interested in increasing their market?
2. I’m not getting seven dollars.
So anyway, Amy, being smarter than I am, answers “yes” to the body wash question. The lady takes us upstairs to where a young man asks us if we have ever taken a survey, explains it, and gives Amy a bunch of papers to sign, including a confidentiality agreement. He did not ask me to sign one, even though I was in the room and trailing around behind them. But because I’m nice, I won’t tell you what it is. I’m sure someone, somewhere, cares.
While she’s reading the paperwork, I talk to him.
Me: So, what’s your target demographic?
Man: Well, you know how we asked you all those screening questions? We’re looking for people between 34-54 who use body wash or bar soap at least five times a week.
Anyway, I decide I don’t want to get the screener in trouble, she probably gets paid chickenfeed anyway. So I don’t tell the young man that the screening wasn’t quite administered properly. Maybe they were on a budget, or maybe she figured I didn’t shower everyday.
Man: We also screen out people who work for Wal-Mart, or the companies which produce these products.
Me: How do you get that information?
Man: We ask the survey takers.
Me: Nobody asked us that question.
Me: Nobody asked us that question. You don’t know where we work.
Man: (after a pause) Well, there you go!
Amy: *rolls eyes* I’m done.
Now things move on to the next phase. She is given a clicker, and pictures of display shelves in supermarket aisles are flashed on a screen in front of us. A camera positioned in front of Amy supposedly tracks her eye movements. I watch the man administering the survey. He is intent on his computer screen, though he continues to explain things to Amy. Have I mentioned he talks to us like we are five years old? Very annoying.
Then we get up and move to another room, where there is a computer with multiple-choice questions. Amy is shown some photos of the body wash in different packaging, and sits down to answer questions. I notice we have been taking the survey now for eighteen minutes, and we are nowhere near done. I wander around before walking into this room, and the original screening lady very sweetly asks me if I would like to wait outside.
“You might get bored. It’s just a bunch of questions.”
Me: (brightly) Actually, I’m terribly interested. I’m an anthropologist, you see.
She: (blankly) Oh. Okay. *shrugs whatever-ly*
I wander back into the room. At some point Amy starts clicking a little randomly. She is bored. Another flaw! The survey is too long. This is something so basic to survey administration, I’m surprised they didn’t think of it.
Finally we are done. Or so we think. The man escorts us into another room, with a mock display unit set up. He asks Amy to locate the product she currently uses. Incidentally, she couldn’t remember it, and when we were done and left, suddenly said, “I just remembered! I actually use their body wash!”
However, she still thinks she uses Dove, so she goes ahead and locates the Dove on the shelf. Then he asks her to locate Product X, their body wash. She quickly scans the shelf and does so.
Man: How did you locate it? Were your eyes drawn to the middle? Did the packaging stand out?
Amy: No, I just scanned the shelf left to right, top to bottom, and stopped when I saw it.
Man: Okay, so the packaging stood out. Was it easy to locate?
Amy: Well, once I saw it it was. Had I been standing in front or it, instead of at an angle, I might have seen it sooner.
Man: Okay, so the packaging stood out.
We gave up. As a matter of fact, I was right in front of the display, and saw Product X right away while Amy was still scanning, so her hypothesis stands a good change of being correct.
Why am I relating this long story? Well, because, as an anthropologist, I have some experience doing qualitative research, and a lot of basic things were being done wrong here. That meant the survey was inefficient, data gathering was flawed, and the resultant data were unreliable. Company X will base its packaging decisions on data like this, which means millions of dollars are eventually spent on data which are not entirely accurate.
Here are the flaws and inefficiencies, as I saw them:
- The screening was not done correctly and potentially introduced errors and confounding factors into the data. I’m merely talking about the questions here, because I don’t have enough information on how they randomized their sample or if there were other conditions prior to randomizing it.
- The screening eliminated potential customers who might have been willing to try Product X were it brought to their notice
- Errors, when pointed out, were not corrected.
- The survey was too long. This leads to the danger of boredom and of the survey takers filling out things arbitrarily just to be done.
- The most glaring flaw of all–not listening to the respondent. As with the last set of questions, the administrator overlaid his pre-defined responses onto Amy’s, thus falsifying data. What he said she said was not what she said at all.
How do you fix these errors? Well, you can and you can’t. All these errors were a result of careless or untrained survey administration. It’s not enough to tell administrators what questions to ask. You also have to impress upon them the importance of efficient and ethical data collection. It seemed to me that the survey administrators were a firm hired by Company X. In this case, the company may not have much option but to trust them to do their job carefully.
What does this have to do with anthropology? Easy enough. Companies need people with the skills to administer and oversee such market research projects. Anthropologists have all the required skills to do this job and do it well. We are trained observers and qualitative researchers. who understand the critical importance of sound data. As an observer, I was able to spot areas in which survey administration could be more efficient and useful to Company X. An anthropologist on their payroll could do a lot more.