Measure

A startup’s job is to (1) rigorously measure where it is right now, confronting the hard truths that assessment reveals, and then (2) devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan.

The development team needs to have good metrics that will prove that the product is getting better. Having more users or even positive opinions from some of them is simply not enough. Actually, if a company does not have proper analytics, the changes in the product may cause harm. It’s because, statistically, most of our hypotheses are wrong.

Three learning milestones.

The first one is establishing the baseline. It’s learning where your product is right now. So, if you build a prototype, you will get the first metrics on all your assumptions. It will tell you a lot about your current stage.

The second one is to tune the engine into the ideal one. As you have all the metrics from the first step, now make it better until you get to the point that you expected the metrics to be at the very beginning.

The third and the last one is the decision - should you preserve it or pivot into a different idea. If you get to the point, that you cannot improve the product anymore to your ideal numbers, maybe it’s time to pivot.

On the early stage of the product building, be sure that you are focused on learning, not just optimization. It’s easy to confuse those two. Optimization is making a better conversion rate on every state of the customer journey. Learning is understanding your customers, their needs, how they use your product, and so on. There is always room for optimization, but what you want to focus on is the big vision and developing the company towards it.

Bear in mind, that you always want to observe the conversion. The total number of users or purchases are not really relevant in the early stage. Instead, you want to observe what’s the increase in the percentage of people who registered or purchased your product. In order to do it correctly, use split testing - show one version of your product to 50% of your users, and the second version to the other half. The groups of people must be the same. Looking at metrics, you will know which version is better. To learn more, read some articles about it on the internet.

The three A’s

There are three important elements of having good metrics.

Actionable - everything that you’re measuring should be an effect of the actions of your team. For example, the total number of registrations is not good enough, instead, you should check the number of registrations per marketing channel with an interpretation of what actions your marketing team did to lead to that.

Accessible - The metrics should be easy to understand for every manager in the team. Very often, companies have complicated reports, that many people in the company don’t know how to interpret them. It’s better to make metrics simple, so everyone can understand them and act on them.

Auditable - Make sure that the data is credible to everyone. Sometimes things don’t go well, and employees must understand why that is.

Principles to follow