How measuring your objectives influences results

One of the things that really fascinates me is the assessment of performance and other metrics. Not just how they are measured, but what effect such measurement has on other aspects of product development, including the completed outcome. This may seem a relatively new approach in Agile, but science has seen a number of takes on the idea that measuring objectives can impact results, and it is these that sparked my interest in the subject.

A great example of how measurement can define what we see in results is quantum physics, where there is a vast discrepancy between what you can see and measure under a microscope, and what is actually happening at the atomic level. Measurement defined our understanding of the sub-atomic until we had the tools to see it, and that is a very clear example of measurement changing how we perceive results.

There are others, perhaps the most famous experiment where measurement explicitly impacts the result is Schrodinger’s cat, while the double-slit experiment is another. In both of these scenarios, whether it is looking in the box at the cat or observing the wave pattern of light, measuring the experiment influences the result of the experiment in a very direct way.

In the case of the cat, by opening the box we determine the actual state of the cat and end the experiment, so the question of whether the cat is alive or dead becomes moot. The problem is, that Schrodinger’s Cat experiment is about how we perceive things, and so opening the box ruins that thought process, it changes the entire process. Similarly with the two slits experiment, by seeking to determine a result, we change the experiment itself.

This can happen in the world of product development too, and our need to measure and assess performance can become an influence on that performance itself, and not always in a positive way.

Understanding how measuring affects us

There is always a need to track progress in any task, but most of the time, whether in science, product development, or any other environment, we tend to ignore the evidence of things like Schrodinger’s cat and assume that such measurement has no effect at all. However, the reality is that it can have an effect, in a number of ways too, and it is this behaviour that we must understand and account for when looking at progress tracking and other needs to assess both products themselves and their development.

Take something relatively simple, a child riding around a backyard on their bicycle. They can do that for hours just having fun, but tell them you are going to time how long it takes to do a lap, and what happens? They start going as fast as they can, the time is the only thing that matters. That is an example of the problems we face when measuring because although we don’t like to admit it, that behaviour isn’t restricted to kids.

If we look at professional life, then you can follow the same direct line of influence. If someone tells you that office printing costs have escalated and everything is going to be more closely monitored, what happens? You print less. It’s a natural instinct, and you may not even consciously make the choice, but you will avoid printing unless absolutely necessary.

Now, that may be a way of lowering office costs, and so in some sense be a success, but what does it really do? Is anyone gaining insight into why printing costs had gone up, or what was being printed that was not necessary?

No. By measuring print use, the task was so changed that it becomes impossible to draw any conclusions about any aspect of the printing process and how it is used within the team. While costs may go down initially as everyone avoids printing, eventually the monitoring will stop, and print use will rise again, but still without any understanding of why it happens.

That is the problem of measurement, a dichotomy of needing to assess a process but also not wanting to influence that process in any way to avoid skewing the outcomes.

A different way of thinking about measurement

Measuring performance across the product development process is crucial in knowing and managing progress across all types of tasks, but they are often used in a way that can actually cause harm to business by forcing teams to focus on a specific target to the detriment of other aspects of the development, or even the product itself.

A good one here is KPIs related to revenue, which at first glance may seem like a necessity, although otherwise innocuous. Revenue is crucial for any business, so it makes sense to focus measurements in that direction, but as we have shown in other, simpler areas of activity, focusing on that one thing, will be significantly influenced.

In this case, while people are focused on revenue generation, this may seem healthy for the business, but not if it is at the expense of other aspects of the development. Revenue generation by itself is not a sign of success for the long-term health of any product and can be alarmingly short-term terminally so in some cases.

A focus on revenue may lead to short-term thinking that seeks to extract as much revenue from each customer in as short a time as possible, which would give excellent KPI numbers of course. But what does it do for the customer? Would they feel like they are dealing with a business that sees them as a long-term partner, or feel like a victim of a cash-grab with no thought about the future?

Most would fall into the latter, and that is not good for business in any time frame apart from the immediate one. But that is what such focus encourages, whether intentional or not. So, what is the answer?

To reassess what it is we are looking to quantify it.

Revenue or profit-based KPIs focus on the end product of the product development process, and as a result force the team to also focus on that, while paying less attention to other aspects of the product, leading to failures of all kinds. The answer then is not to stop measuring, but to measure the things that deliver that end product. By increasing focus on those things that drive revenue generation, rather than revenue itself, we build success by continually pushing for more of the actions that are generating success.

If we think about any product development, what is a good indicator that it is succeeding, not just today but in building a platform for the future? Revenue, as we have shown, is not it, neither are efficiency, expenditure, or other fiscal measures. If you were to isolate one measurable aspect of a product that if maintained at high enough levels would ensure continued success that would be customer satisfaction.

If a customer is satisfied, they are much more likely to come back and make repeat purchases. They are also more likely to encourage their friends and families to do the same. By maintaining a high level of customer satisfaction, we offer the best chance of keeping that cycle going. More custom and a growing audience also lead to increased revenue too, but customer satisfaction is also a great option for measurement now we understand the impact of measurement on outcomes.

As we have looked at the focus on revenue, which can lead to unhappy customers and ultimately a loss of revenue as a result, if we apply that same behaviour to a customer satisfaction KPI, we get a different outcome. With customer satisfaction as the focus of measurement, the team will transition to ensuring that customer satisfaction remains high. The difference is that, unlike revenue measurement, there are no negative consequences for doing this.

All the measures needed to keep customer satisfaction high boost the development itself, and with customer satisfaction kept at a high level, then revenue generation is also boosted. No aspect of the measurement harms the overall product development process.

That change can transform any product and continue to deliver success without losing out in other areas. It is a small change, but one that should be embraced throughout any Agile development process today.

Rethinking the way that we assess our goals

So, we must not abandon measurement, but seek to approach measurement in a different way, one that focuses on the driving factors of our goal, rather than the revenue or other metrics that represent it.

In this way, we avoid the problems of measurement that impacts our outcomes, but by choosing carefully, can find measurements that boost overall performance by being focused on as well. This is the strategy for measurement that we must adopt that meets the needs of oversight and of product development success.

Customer satisfaction delivers this, but is not alone in both offering effective insight and driving the behaviour and outcomes that we are looking for from a product. For instance, during the launch phase of a project, the focus should always be on knowledge acquisition, as that is the key foundation that which everything else is built.

Again, by putting the focus on knowledge acquisition, what is it we are encouraging? More knowledge acquisition. This is beneficial for the product as well as provides an effective metric for development progress. It is this approach, as with customer satisfaction in later-stage projects that serves to deliver useful measurement but also encourages positive behaviour that brings benefits too.