1. Survey Questions
We ask a set of questions that are based on decades of consumer product development and meant to triangulate strengths and opportunities for improvement at the idea stage. Questions cover the following areas:
- Use Intent – Willingness to use, download or otherwise try a new idea; where applicable we ask Purchase Intent instead or in addition
- Liking – The extent to which people like or dislike the innovation
- Value – To what degree people feel that the new product or service adds value to their lives (given the price, if applicable)
- Uniqueness – The extent to which the idea is new-to-the-world
- Believability – The ability of the idea and its description to convince people that it will deliver the promised benefit
- Usage Frequency – How often people expect they would use the innovation; usually ranging in degrees from daily to yearly
- Need Fulfillment – To what degree the product or service solves a problem or fulfills a need
- Word-of-Mouth – Respondents’ willingness to tell friends and family about the new idea
Of course! Questions can easily be added to an existing test. Most choose the Screener Plus test to add 5 questions to the survey, but you can customize your test with more or fewer questions. These questions can be in the form of Multiple Choice (select one or more than one), Rank Order, and Open Ended (fill-in-the-blank).
You can use the MVC Test to get customer feedback on any decision that you are considering. However there are nuanced ways to get the feedback in more useful ways. In pricing, for example, one should not simple ask people whether or how much they would be willing to pay, since they will always say they prefer free or the lower cost option. So on pricing, we recommend using a Van Westendorp test to better assess how high or low they will go. Meanwhile, we often recommend forced ranking questions to understand which features people most desire.
But you’re not on your own! Our experienced consultants will work with you throughout the project, from setting up the right questions to interpreting the results.
2. Customer Panel
Every research panel consists of a group of people who register, provide information about themselves, and agree to complete surveys in return for some kind of incentive. They typically are alerted via email when a new survey is available, and answer a few initial screening questions to see if they qualify for the research.
But some panels are better than others. We use a specific outside panel supplier with access to over 4 million people to conduct MVC Test research. We judge quality of the panel by the consistency and quality of panelists’ responses. For example, in our open-ended (e.g. fill-in-the-blank) questions, the answers panelists provide are rich and insightful. You will find few “junk” responses where people are obviously rushing through to get to their incentive, and we have ways to kick out bad respondents and reward high-quality feedback.
In our history of running hundreds of MVC Tests we have seen that respondents usually reply honestly. They are used to completing surveys and trust that their privacy is protected. We also choose to work with a specific panel supplier that has several approaches to encourage honest answers. It uses “gotcha” questions to ensure that people are answering correctly, and we have the ability to flag and remove panelists who provide junk responses.
First, there is a very low chance that one of the few hundred panelists who complete a survey will capture and share your idea. Our panel supplier ensures that its research subjects agree not to copy or share what they see. Panelists only see a short description or video of the idea on their screens, and it is difficult to capture and copy an entire business model based on this limited information.
While there is always risk that someone could capture a screen shot or jot down a description of your idea, the benefits of getting customer feedback far outweigh the tiny risk of your idea leaking out. Even the world’s largest companies with billion-dollar research budgets know that they must similarly expose their ideas and learn from customer feedback using the same panel we use.
We have run enough MVC tests to see similar ideas pop up continually, which suggests that ideas are important, but winners are those who best learn, adjust and execute in the market. If you’re the first one to develop and MVC Test your idea, you’ve got a huge head start!
Today, most of our tests involve B2C innovations for a few reasons. First, it is much more difficult to effectively gauge potential for a B2C innovation to succeed without some broad, nationally-rep research. B2B startups, on the other hand, usually have an easier time learning enough from a small group of potential customers in one-on-one research. Second, the more difficult it is to find a specific niche group, the more expensive research panels are. For example, if you wanted to do online research with a nurse, it could cost $30 per person. These costs are out of reach for most startups.
However we are continually working to bring MVC Testing to B2B innovators and we often do have options. Just let us know what you are looking for and we’ll do our best to come up with a solution.
3. Idea Description
For most MVC Tests we work with you to create a written description of your idea, which is also known as a “Concept”. The concept looks like a very basic powerpoint print ad. It might sound basic, but that’s the point–the purpose of the concept is to describe your idea in a straightforward way. There is an art and science to concept development that is based on what billion-dollar companies have used for decades. Survey participants will be exposed to the concept in full screen mode and click to the questions once they have read it. The MVC Test team will work with you to build the concept, which includes the following elements:
Problem Statement: Short summary of what you believe customers’ need or pain point is.
Benefit Statement: Another sentence that introduces your idea as the solution and describes the job that it does.
Reason to Believe (RTB): A sentence or two that describes how your product works, why we should believe it’s true.
Other Info: A few other key details that people need to know to make a decision to use or buy your product (e.g. price, packaging, operating system)
Screen shot: You might want to include a mock up of the screen or product package. Or include some kind of relevant stock photo. This is just a secondary cue for readers, so don’t go designing something special just for this.
Take a look at the example below for an app called Phogo:
Either option is sure to provide useful customer development feedback, so this is more of a question of your time and budget. The sight, sound and motion of a 2-minute video can better summarize your idea, but there is no reason to hold off on learning and spending on a video just for this test. If you believe your idea is too complex for a written description, then you may need to simplify your core proposition. After all, you cannot force every potential customer, investor or partner to watch a video!
Remember, the purpose of the MVC Test is to simply and quickly get a number of responses to a description of What your idea is, How it works, and Why it is right for your customer. In the real world this may happen through a website, TV commercial, news article, or one friend talking with another. While a video can communicate more details, the MVC Test is focused on the core benefit and customer demand. We have seen no significant difference in the learnings that come out of a written vs. video description.
The concept description framework described in the question above has been proven to succinctly describe hundreds of ideas across dozens of product and service categories. However survey respondents are sometimes confused about how the tested idea works or the description seems unclear. This is actually one of the benefits of MVC Testing–you are able to get feedback on your idea and learn what is confusing before it goes on your website or gets in front of an investor.
There are a few ways that MVC Testing can help you decide on the best possible positioning–or language to describe your idea. First, you can use a broad description of the idea, then use survey questions to understand what people like most about it. This often helps companies focus their message on the maximum impact. If there are multiple broad descriptions that you want to learn about, then we can help you run two MVC Tests in parallel and use the output to see if one of the options results in better scores than the other.
4. Database Comparison
If 15% of survey respondents say that your idea will Definitely solve a problem for them, is that a good score or a bad score? It’s hard to judge such a number on its own. However when we see that this score is in the Bottom 20% of all MVC Test results, it suggests that an idea is not addressing a big unsolved need for most people. Overall, the database comparison helps us to triangulate the strengths and weaknesses of new ideas, as often patterns arise when we layer answers to our eight concept assessment questions against the database averages. Since each startup idea is competing for mind share among consumers, high or low scores compared to this database can help predict ability to break through the clutter and beat the odds. And because we have tested so many types of startups, chances are that we can provide more specific comparisons within your product or service category.
Our database consists of a wide variety of digital products and services such as apps, websites, and devices. We continually run new tests and adjust the numbers every week. We have tested ideas in a wide range of categories, including mobile, social, travel, parenting, e-commerce, healthcare, athletics, nutrition, entertainment, fashion, shopping, gifts, coupons, and advertising. Increasingly we are able to not only share comparisons to the total database, but can share how your idea compares to others in a similar category. For example, we can show how your new travel-related e-commerce idea compares to three other, unidentified travel+e-commerce companies we have MVC tested in the past year.
5. Sharing the Results
As outlined in the Professional Services Agreement, we will never disclose any confidential information about your idea that we have received access to through the MVC Test process. This includes, but is not limited to, your business description, market strategy, and MVC Test results. Your test results do go blindly into our comparative database, but specific results are never shared with outside parties. We will never speak directly about your results to an investor or partner without your permission.
Of course! The output of the MVC Test belongs to you and can be used in anything from investor pitches to media interviews. Our clients often use the consumer data that comes from the test to show the size of the potential market and prove that customers are highly interested in their solution. We always include a description of the MVC Test service and process in our reports to help educate these partners, and we’re happy spend extra time working with you to package the test output into a pitch.
Poor results do not necessarily mean your idea is doomed! Whether the results are high, low, or (mostly) in between, we will share insights behind the reactions and offer suggestions for improvement and next steps. Successful innovation is about a continuous cycle of learning and improvement.
As for sharing with investors or partners–if you’re paying the bill, you own the results, and are free to do with them what you will.
6. Prices & Timing
Once we have agreed on the idea description and test questions, timing varies according to the complexity of your test. The actual panel results are typically back in 24-48 hours, and then our team works on the analysis and recommendations. The Screener and Screener-Plus tests are usually back in less than a week, while the Full test may take up to two weeks. If you have a rush request, we can usually accommodate your timing need at no additional charge.
We start with one of the three options below and adjust based on your specific target audience and business needs. The Screener Test is our starter offering; it exposes 150 respondents to your idea and returns with answers to our core MVC Test questions and a comparison to our database. The Screener Plus test adds open ended Like and Dislike questions, as well as five additional questions of your choice. The Full test expands the response pool to 300 people so that demographic and psychographic differences are more likely to be visible. This option also includes additional questions about respondents current habits and solutions, and we include an Engagement Forecast™ that walks through a market sizing exercise with predictions of likely users and revenue potential.
This type of service typically is priced at tens of thousands of dollars by large research firms, but our starting point was to create a research service that fit the budgets of unfunded startups and other lean innovators. Our prices are determined by two key costs.
Survey Panel – This is a hard cost from our research partners. These costs vary according to how many people are needed to answer, how many questions they must answer, and how hard it is to find them.
Analysis & Consulting – This covers the time for our experienced team members to work with you to set up the test and create your idea description, then performs analysis and presents recommendations.
As for value, that’s for our customers to judge. Even our most expensive test is a fraction of the cost and time to develop and launch a Minimum Viable Product. And by running a MVC Test before developing, you can significantly improve the direction of your MVP dollars. We’re cheaper than an attorney, and can provide the early insights that meaningfully improve your odds of success for years to come.
Unfortunately each survey comes with its own incremental costs. Every time you run a test we are charged to get feedback from a new panel of customers, and we have to do new work to analyze the results and share recommendations. However we sometimes offer discounts to returning customers when they are testing a similar idea. It never hurts to ask and we’ll see what we can do!
Unfortunately we do not provide free testing, as lovingly described in this blog post. However it costs nothing to use our concept creator to build a description of your idea and approach potential customers on your own. This is actually the best way to start the process of Customer Development, and it’s exactly how the biggest companies in the world start getting feedback on their ideas. Once you have shared your idea with a few dozen target customers in one-on-one conversations, you will have a tremendous amount of feedback to use in refining and improving your idea. When you’re ready for broader customer feedback and a comparison to a database of other ideas, we’ll be here for you!
We prefer to keep our everyday prices very low to meet the needs of startups and lean innovators. However we sometimes share discount codes during events and special occasions. And since you made it this far, you can used the code “FAQ” to save $100 on your first test with us.
7. For Doubters
Certainly lean innovators are used to looking for the free option first, and very low-cost survey tools abound and even Amazon and Google are getting in on the act. But “you get what you pay for” holds true in market research, as poor-quality data can be unusable at best, and steer your company down costly decisions at worst. Friends and family are usually willing to help out, but they are pre-disposed to like your idea, and might not really be in your target audience. Sometimes market research feels like the early days of web design–anyone can do it thanks to low-cost tools, but there is a science and art to the discipline.
We believe we add high value for the money in two ways: First, we use a high quality panel of several million U.S. consumers. We choose not to maintain a panel ourselves so that we can continually shift to the highest-quality, trusted responses at a fair price. High-quality responses ensures that respondents are not simply racing through questions to receive their incentive. You will see more well-written responses to open-ended questions that help you get a feel for what respondents are really thinking.
Second, we provide guidance and recommendations from consultants who have spent years launching new products of their own. Our consultants work with you to make sure that your idea description is focused, you are targeting the right audience, questions are well-written. No other survey tool (free or otherwise) can provide a comparison to other startups’ test results. And your final report comes with recommended areas of improvement for your product positioning, marketing and business model.
As Jason Anderson says, “Information is free, insight is expensive, and action is priceless.”
The MVC Test is not primarily designed as a predictive tool–rather as a way for innovators to learn from customer feedback and adjust their product or service as early as possible. A company that scores very well will find insights that help them perform even better, and a low-scoring idea offers founders the chance to pivot before they have to.
However we are continually gathering in-market data for every company that we have tested. In the months ahead we expect to publish results showing that companies with high MVC Test scores are more likely to have continued to survive, secure additional funding, and growth both users and revenues.
It is never too early or too late to use market research to learn and improve your business. We often work with companies that have launched but are still eager to learn. They may have seen user growth stall, are in the process of seeking investment, are considering a pivot, or are interested in broadening to new markets or customer targets. Our team will work with you to understand your specific needs and craft a testing plan that will help you move forward no matter what situation you find yourself in.
Now that’s a good question! But seriously, the number one reason that startups fail is that they are not needed enough by their target customers. For a fraction of the time and money of developing a minimum viable product and waiting for customer feedback to trickle in, you can get invaluable customer feedback in a proven research methodology and recommendations from a team of successful innovators and investors.