Indicators, targets, benchmarks – sorting out the terminology

Many different terms are used in the outcomes and performance management area for measurement and indicators. Often there is considerable confusion about these terms. The short definitions I use from outcomes theory are:

• Outcomes – causes or effects in the real world. Whether or not such causes can be measured is a separate issue (see previous blog for features of outcomes)
• Steps – lower level causes which lead to higher-level outcomes. Because causal processes reside in causal hierarchies, outcomes at one level can be steps for achieving even higher-level outcomes, therefore to refer to causes and effects at any level the general term ‘outcomes and steps’ is used.
• Measurements – measure whether or not an outcome or step has occurred (or how much of it has occurred).
• Indicator – a measurement of an outcome or step.
• Target – a level on an indicator.
• Benchmark – levels on indicators already achieved by other players, or by the same player at an earlier time or in another setting.
• Priorities – an outcome or step which is thought to be the most important for a player to focus their efforts on changing.

Indicator Problems stressing out US Patent Office examiners

In an earlier blog posting on the banking system, I commented on the problems associated with indicators which can be distorted by employees. You can either use such indicators to get an accurate measure of an outcome or use them for incentivizing employees, but not both. Of course, if they’re being distorted by employess to maximize their bonuses, then the wrong types of behavior are probably being incentivized. A different problem can arise in those situations where it much harder to distort indicators. These situations in which it’s relatively easy to independently verify indicators are described in the social sciences as being situations where such indicators are reliable. However, indicators need to not only be reliable, they also need to be valid – they need to actually measure what they claim to measure. The US Patent and Trademark Office is having indicator problems of this second type with their production quotas. According to a Washington Post article their production quotas have not been adjusted since 1976 and modern patents are more compex and therefore take more time to process. As a result 67% of staff see production quotas as among the top reasons they would consider leaving and the office has a turnover crisis according to the General Accounting Office.

Trading-off indicator accuracy for incentivization – forecast calls for pain

An article in the London Times today (4 October 2007) – Why a bulging bonus is part of the problem, argues that the current credit crisis following the melt-down of the US mortgage markets has not yet run its full course. In particular, the problem is that in the global banking system, ‘the losers do not know that they are losers, even now’. It will only be when banks start selling off their investment assets that the real price of these assets will be determined and at that time there’ll be lots more pain. In the meantime, it’s likely that a falsely positive picture will be painted because of the bonus systems for employees in the banking industry. This system means that ‘there is tremendous incentive for everyone from the chief executive down to believe the best when millions in personal rewards are at stake’. As a result they’ll overvalue banks assets in the short term. In the run up to the crisis, the first half year results for banks were the best they’ve ever achieved. In order for employees to get an annual bonus they’ll need to also show good results in the second half of the year or else they’ll lose the potential bonus they have on the books from first half of the year.

This is a classic example of the outcomes theory principle of the trade-off between indicator accuracy and using indicators for incentivization where indicators are easy to falsify. And ‘it is the very nature of the bonus system that may end up prolonging the agony’. Perhaps we could go even further than the article and wonder about the roll of the bonus system not only in prolonging the agony, but as a contributing factor in the creation of the problem in the first place.