Throughout our educational lives from grade school through university we test to prove we have mastered a topic. In engineering school we even back test our hypothesis to prove them accurate predictors of future trends. Given this it is understandable that testing for failure is tough to learn.
The images of success are the first electric lightbulb providing a steady warm glow or the voice over the telephone “Watson, can you hear me?” Today we strive for those 3.5 or better stars in our app rating still focused on getting a pass to the next stage of the lean framework. Looking to move past questions or concerns not embrace them.
Working with product teams it is culturally challenging to learn the balance of getting it right and learning the failure points. We have all had the interview questions “what are your strengths” “what are your weaknesses” my challenge is how we answer those same questions for our work output as product managers. What are the areas of concern? Collect issues from your test like a roadmap in the future, “Release ready” is a choice not a grade, it is your choice. Form great questions ahead of time for all your experiments, wait for feedback and clarify what you see what you hear. Clear communication is your responsibility.
A well designed test asks questions, watching the user explore the product rather than the documentation. Finding the tricks as someone would in a video game is not what a test is interested in as much as where they go in the search. Create tests that test single performance items, causality can obscure results. Create questions and think If so, so what? It is a potential trap for product professionals as we are problem solvers, we want to jump in and fix. It is painful to watch our work struggle.
Some may say, I enjoy asking questions to much but for me the process of discovery trumps the answer and if I have an idea in my head I am eager to be wrong. This is the difference between a learner and a know it all. Learning is about the answer you get, not telling the correct answer or having proof your idea is the correct one. Getting another to agree with you is not a test, getting your prospect to learn how to use your product through painful steps is not success, learning how their intuition works with your product is the test.
I like to tell the story of the US and USSR rocket programs, in the US the modeling and test work was mostly done through simulation and micro test. IN the USSR they built it and blew it up, then built it again and pushed it to complete failure. The result, the USSR rocket engines are still the core of space launch as a result of their robust performance, the US engines are a risk and have a documented launch failure history under heavy loads. The message is don’t be afraid of blowing it up once in a while but always learn making incremental changes.
Strive to walk away from tests with new data, with new insights, not a pass go card. This is not easy trust me, focus groups and early trail customers want you to succeed. Frankly they are more often giving people or they would not be trailing your MVP, investing their time also. The platitudes are great to hear but should be taken with the mindset that I learned from my first sale course, “Customers lie three times”. This is not a negative, it is more positive about people’s good intentions and your need to learn weakness of your offering.
Test to failure and that means at times finding the failure that is unspoken.