Here's a situation you may have experienced: You've decided to run some online consumer research for your new product idea. You feel confident because you chose to test with a research company that a lot of big companies have used. The data comes in and looks great--over 50% of people say that they would "Definitely" use your product. However the results from one question are odd: On "What do you like about this product?", where people had to fill in a blank, a large number of people wrote in things like "ajfk", "n/a", and "IDK". Many others gave one-word responses such as "easy", "good" and "free". Oh well, you think, at least the charts and graphs look good. Time to open the champagne, right? Maybe not. Despite the misunderstood words of Steve Jobs, market research is experiencing a bit of a resurgence recently with the growth of low-cost online research options that make it easy to launch tests and analyze results. SurveyMonkey was recently valued at $1 billion, and Qualtrics turned down a $500 million buyout offer. Two of the biggest players in today's economy, Google and Amazon, are playing in the research game now. Google Consumer Surveys offers feedback for as cheap at 10-cents per response, and Amazon's Mechanical Turk business can do the job for as little as 2-cents. At that price, there's no need to nag friends and family to fill out your surveys! Suddenly market research that used to costs thousands of dollars and take weeks to complete is available for pennies in comparison. I count myself as one of the "new turks" of market research. I could never have launched the Minimum Viable Concept Test without a low-cost, self-serve market research partner, which allowed me to provide customers with actionable feedback on their ideas in a couple of days for as low as $999. But after spending my career conducting research with over a hundred suppliers and seeing the answers of hundreds of thousands of research subjects, I've learned that there is still an art and science to practicing market research. And just because Google is getting in the business doesn't mean its output is high quality. Google also created a free blogging platform, after all, but that doesn't mean everyone who opens a Blogger account can write worth a crap. You might not have realized that the rise of online research has also spurred a rise in online research cheaters. You see, all of those new research tests that are enabled by low prices need actual people to fill them out--and a lot of those new survey-takers are only in it for the money, gift cards, or "chances to win" that are promised for a few minutes of their time. In fact, over at YouTube you can find hundreds of videos teaching you how to fill out surveys with as little work and thought as possible. I know from testing several different research panel providers that this is a big problem. I have tested with most of the biggest players in the market, and I've found that many of them provide dangerously poor responses. I use the word "dangerously" to describe two scary situations for a business leader: At best, poor research quality will waste your limited budget and delay your project if you are forced to start over. At worse, you may act on the data you receive--confident that the customer has spoken--and find that the real world doesn't match up to the results you received. Your career and life savings might be at risk because you trusted a large company to deliver quality feedback. So, you might be wondering, how can I tell if a market research panel is of high quality? It's actually pretty simple: Include some open-ended questions in your survey and see how well they are answered. "Open-ended" is another word for "fill-in-the-blank" questions (or, if you're a market research nerd like me, "verbatims"). This is where people cannot just pick a multiple-choice answer, and instead have to actually type something in. If you get a lot of "junk" answers like ";lkj" and "n/a" or even those one-word responses such as "good" and "like" then you need to go back to your survey company and demand that they either re-run the study with a better panel or, better yet, give you a refund and take your business elsewhere. You may not know that most research companies outsource their panels to other suppliers and only pick based on low prices. Google's new service doesn't include open-ends yet, so I'd stay away despite their claims of accuracy and discount codes. Open-ended questions not only help you gauge the quality of the research, but they are often the most helpful responses you can get. After all, the reason you conduct research is not to "get good numbers", but rather to understand what's going on in people's heads--and there is no better way to learn this than reading reactions in their own words. Even when I worked at Procter & Gamble and did research with 1,000 customers in a high-quality panel, the first thing I would do when research reports came in was to read through every single open-ended response. In the MVC Test, we provide clients with this raw output, and code the comments by hand so that we can see what people said most and least-often. I'm proud to say that I have had tremendous, positive results in working with my panel supplier, Ask Your Target Market (AYTM). I first came to AYTM for its good prices and terrific self-serve solution, but from my first test I was blown away by the quality of its panel responses. Here's one example of such high-quality responses we saw when we ran a test for Roadtrippers, which I described in a previous blog post:

If I were to use it, I like that it picks the top options only. I know personally when I research stuff I only even consider the 4-5 star ratings. Having everything they offer, and for free, is a great thing to consider if I take a long trip."

This is a lot more helpful than "IDK." Over the past year I've gotten to know Lev Mazin, CEO of AYTM. He has taken me through the tools he uses to make sure that his panel is high quality. There are gotcha questions to catch cheaters, and clients can either "thank" good respondents or "flag" the few bad actors that slip through. You see, Lev knows that market research is meaningless (and his business is doomed) if his clients ever doubt the reliability of the surveys they conduct with him. That's great, because I, too, must stand behind these results for my business to succeed! I do believe that the future will be brighter thanks to cheaper, faster and easier ways for us to bring customer feedback into our innovation processes. But if success was that easy, fast and cheap, then everyone would be successful. As Jason Anderson recently wrote: "Information is free. Insight is expensive. Action is priceless." You should learn more about the MVC Test here, and follow me on Twitter here. +Bob Gilbreath