By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
Validation is probably the most important, most frequently referred, and yet the most neglected part of a startup’s early-stage journey. Even the startup founders that try validation do it half-heartedly or without learning the art and science of it. This blog post intends to sum up all the typical mistakes, the science behind them, and our proposed path to validation.
What should you be validating?
Startup founders often mix up the validation of the solution (and the original innovative idea) and the validation of the problem.
What is the difference between the idea, the solution, and the problem?
The idea is the value creation part. The things you would like to change in the current settings. It’s very hard to validate or measure the idea. It’s more like a vision. Can you foresee that the way you want to change things is going to be the predominant approach in 5, 10 or 20 years?
The solution. Validating a solution is not a one time action. It’s a continuous set of actions under the customer or user experience umbrella. Testing how your solution works, whether it fulfills expectations and is used in the way that creates value and correlates with your vision outlined in the idea validation.
Problem validation. This is what you should be doing at the beginning of your journey. Do people have this problem? Is it at the forefront of their thought process? Is it a top 3-5 problem? How hard is it to trigger spending for this problem? What is the alternative solution? What are they spending money on now in that problem space?
Note the difference to understand the strategy! Idea, solution, problem.
Problem validation is the only thing you should focus on at the beginning. That’s what “venture” capitalists are looking at. Few VCs take the risk of buying into the vision and are more likely to invest in solutions that have a strong problem validation. Maybe even the problem you are solving is so apparent that they don’t even ask for problem validation results.
The reason why this is so important is because it is almost impossible to validate an innovative solution, especially if no competition exists. Alternate solutions are always on the market as a problem is solved one way or the other. This doesn’t mean that you have direct competition that you can clearly compete against or compare your solution to.
If a problem is not on the top of the customers’ mind, there is a serious need for marketing & education efforts. These two money burning machines are the vessels of changing priorities and the thinking patterns about the problem space. But there is a challenge: you are not Apple, Amazon or Microsoft. You don’t have all the time in the world or the money required to do the mindset change. Time and money are the two most important assets of thought transformation a startup doesn’t have.
Exactly because of these limitations (time, money and problem space) is a prior problem validation so important.
Now we know why you should do validation and what you should try to validate. Before we jump into our best practices, let’s see how you usually do it and why you shouldn’t.
There is a reason we didn’t mention surveys and validation in the same paragraph up until this point. Surveys are mostly useless when validating a startup idea. They just don’t go along. This is rule number one. And probably the most important one.
As with all rules, there are exceptions, but I haven’t seen any in the last 10 years. A successful startup that has used surveys as the only validation? Nah.
The key problems with surveys.
Filling a survey is not the same as making a purchase. It’s not a real-life buying scenario. Customers in reality vote with their money. How they spend it, where they spend it. Either for your solution or an alternate one. People are happy to tell any kind of bullshit when filling a survey but when it comes to real-life transactions, they lose faith at the counter. They tell you anything you want to hear in the survey then spend exactly zero dollars at the end.
Helpful people. Anyone filling the survey is trying to help and support you. People who will end up buying your stuff are not supportive of you. They do not and should not really care about you as a person. They don’t want you to feel good. They are looking for stuff that makes THEM feel good. Therefore people with personal supportive motives in the surveys have absolutely no value when you evaluate the market.
Wrong audience. Those who are filling the surveys are often not your real target audience. This is most prominent in a B2B setting but we still see the “400 people were supporting the idea by filling the survey and giving us positive feedback! This means there is a vacuum on the market!” It’s like a drug addiction for founders. How do you know who filled it? Or that they have money? Or that they make decisions about money?
I personally love a lot of stuff, would love to spend money on lots of stuff, have plenty of problems that need to be solved, BUT if it isn’t me calling the shots over our family budget, I am a no one. This is the so-called non-decision-maker error/bias. Another personal example of this is asking me about dairy products: happy to answer, but I am on a lactose-free diet - relevancy issue.
Bad survey design. Market research is a profession. There is a reason for that. If you are not a professional, chances are high that your survey design will be bad. Full of unintended influences, prone to errors. A great survey uses multiple, similar questions from different angles to gauge the same topic. This is used to filter false positive or false negative answers and to check the coherency of the surveyed.
Trolls. Survey Trolls. Take me as an example again. I am the biggest NPS (net promoter score) troll. Whenever I am asked a net promoter question (and I didn’t have a 10/10 experience) I just pick a random answer around the threshold values. Or if a survey starts to bore me, I just pick answers fully random. I know that if I always pick the first answer, it will be easily sorted out, so I don’t do that. Just a random pattern. Sort it out if you can.
A friend’s story in the matter: After a survey, they realized one of the biggest demographic group turned out to be 90+ year old female miners. Why? In the local language, those three characteristics were first the possible options in their respective drop-down lists. So people trolling or not liking the share of details just clicked the first available.
The good survey is long. And short. You read that right. It’s not a grammar mistake or an oxymoron. The shorter the survey, the more people will fill it, the more data you will have. The longer survey can be well designed and might provide useful information. The short, professionally designed survey doesn’t exist. Therefore a well designed and at the same time useful survey (for startup problem validation) doesn’t exist. Sorry.
No free text entry field. I can say with 100% certainty that not all answer options will come to your mind during the survey design phase. Trust me on that one. And you will just omit the free text option. Why would you since you have no hope of doing anything with that data if you get 400 or even more responses? Still, you are adamant that the answers or better, the preset options include the real answers of potential customers.
Behavioral Economics. This is where I should have started probably. During the last decades, it became apparent that we make decisions highly influenced by external stimuli. Or better put, even if unintentionally; all choices and all context have unintended, decision-bending behavioral influences. Some by design, some by fallacy. A primer on this topic is Dan Ariely’s Predictably Irrational. Quick and entertaining read on the most important research in the field. Your survey will contain lots of biases. You can try to correct these, but think about all the biases that will still be in effect and you have no control over them.
Despite all these challenges, there are situations where a survey is useful:
Quantitative research on numbers: Age, experience, factual data which are easy to recall and are not opinionated. Demographics on your existing customer base.
Surveying existing customers: Their response rate will be adequate even if the survey is well-designed, therefore long and detailed.
Conjoint analysis: When you are trying to figure out the right package to be offered or gauge the set of features that will be the most desired. Another profession within the profession. You might see it as the knowledge that can be absorbed in a weekend. Your call.
How to do validation properly
The only method we believe in is doing interviews. Face to face or online, or even with phone calls. Startups fail with interviews because they have preset questions instead of guidelines and they don’t go with the flow of the conversation.
During an interview, your goal is to understand the decision-making process and priorities of your potential customer. It’s a bit of reverse engineering. You should enter the interview with a blank canvas and no judgment on whether the person or the target group represented by the individual will be your perfect niche. You just have an open conversation about the problem space regulated by important guidelines.
Guiding principles for validation interviews
No preparation for the interviewee. Never tell the interviewees what exactly the discussion is about. Just tell them you are looking for stories on a specific problem space.
Never tell the exact goal in advance. The interview always consists of two parts. Before and after revealing the exact idea or problem you are trying to validate. People are always trying to help you and pinpointing too many details will influence their answers. And that’s bad for you in that phase. Don’t let it happen.
Don’t have preset questions. Preset questions are about you and not the person you are talking to. The validation interview is a conversation. A free-flow conversation between a (mostly) questioner and the person questioned. Be curious!
Look for stories. If you wanted one secret sauce from our vast library of innovation secrets, this would be it. Most of the decision-making happens on an emotional level and stories are the best representation of this emotional map. Not to mention that when done properly, stories are the best representation of truth. The truth of the customer. It doesn’t matter if it is a factual truth or just a subjective memory because this is the playing field your service or product will compete in.
Create an emotional baseline. Evaluating the interview is based on emotional spikes against the baseline. If you jump right into it, you will have no time and no chance to establish this baseline. You will not know what was important to the interviewee and what they only said is important.
Always give them a chance to figure out the actual goal. Be prepared to lead them into a dead end. People love puzzles. Your research is a puzzle to them. They should believe that they are in the right domain and got you figured out. For this to work effectively, prepare a research goal that you can use as a distraction. By guiding the interviewee to believe in this direction you help create the baseline and steer the conversation later to the domain of your interest.
This is not a UX interview. We don’t care how current solutions are used. We are measuring if there will be a buying intent, and how strong the problem perception is. E.g. how easy it will be to create a buying trigger.
Make it short. The best validation interviews are 10-15 minutes long. Anything after that is not necessary. Remember, you are trying to figure out what the problem perception is. How important the targeted problem is. If you have to dig for 25-50 minutes, then they don’t have a burning problem - you would need to heavily invest into door opening during sales to face that depth.
Stay as long as you can. Although from the buying perspective you will not learn much more from the 15-minute mark on, you can still learn a lot about what your future solution should entail. Objections and context. And respect the time of your counterpart. If they are spending time with you and enjoy the conversation, don’t be rude and never tell them, ok, we can stop here, I had enough. Be respectful even if the interview is just another dead end.
Don’t pay or reward the interview directly. It is an influence, a significant one. People will try to help you if they get rewarded. It can become extremely hard to remove this anchor and get honest feedback. In our experience, most people are willing to help you if it takes 10 minutes max without any additional reward. Just for the joy of being part of a bigger story or in the end, learning about new innovation and drawing inspiration from it.
The interview is the reward and a great place to get serendipity working. Respect the people spending time with you by telling about the innovative idea. They might connect you with the next person to be interviewed or a potential lead, investor. Or they might become one of these.
All of the above are guidelines and not rules. You might want to change this according to your situation or based on the behavior or answers, stories of the interviewee.
What’s next?
The result of such interviews is that you will have an understanding of the behavioral descriptors of the perfect niche and you will start seeing patterns in how these descriptors are more representative in one demographics group than the other. Or how certain demographic qualifiers are more representative for one behavior group than another. After you have the behavior-demographics connection, and only then can you start doing desk research to try to assess the size of the potential target market or the so called TAM.
In case you want a free (as in beer) mentoring on all topics startup or innovation, book a session with us here.
