When Knowing Becomes Expensive

A software supplier is weeks away from delivering an infotainment system for a six-figure luxury car when a stability analysis lands on the table. The news is not good. This isn't a matter of a few remaining bugs or some rough edges to polish before launch. The analysis suggests that ordinary customers may start experiencing serious performance degradation and crashes after only a day or two of active use.

There is still a choice. The supplier can delay the delivery, miss the milestone and take the consequences immediately. Or it can deliver on time and deal with the quality problems later.

Now add one small detail: the people making that decision are measured on the delivery. Their targets depend on it. Perhaps their bonuses do too.

Suddenly, the analysis itself has become a problem.

We usually think of incentives as something that changes behavior. Reward people for achieving something and, unsurprisingly, they will try harder to achieve it. But incentives can do something more subtle and, in complex organizations, potentially more damaging.

They can influence what the organization gets to know.

Years ago, we were asked to analyze the stability of an infotainment system being developed by a supplier for a luxury vehicle manufacturer. We knew very little about the project schedule, but what we saw looked spectacularly bad. So bad, in fact, that we made what seemed like an obvious assumption: this had to be an early development build.

And that would have been perfectly fine. Software under development is supposed to have problems; that's why you test it. Perfect software doesn't exist, and anyone who has worked with sufficiently complex systems knows that shipping always involves trade-offs.

You can even enter production knowingly with significant bugs and still be making the right decision. Production capacity may have been reserved months earlier, the problem may be well understood and a fix may already exist. With connected products, an over-the-air (OTA) update can sometimes be waiting before the customer uses the product for the first time. Stopping production under those circumstances might be a considerably worse decision than shipping.

That's risk management. What we were looking at felt different.

Even with limited analysis coverage, we found serious degradation, crashes, application restarts, resource problems and system reboots. Our analysis indicated that significant performance problems and crashes could become visible to an ordinary user within a day or two of active use. It was, by a wide margin, the worst automotive software we had analyzed.

Initially, none of this was particularly shocking. We thought there was plenty of development left, and finding ugly problems early is exactly what you want.

Then came the review meeting.

We presented the findings and our recommendations. Somewhere during the discussion, however, the supplier mentioned a rather significant detail nobody had thought to tell us: this wasn't an early development build. It was due to be delivered to the original equipment manufacturer (OEM) in a matter of weeks.

It was going into production.

The response to our analysis was essentially: Interesting. But we don't have time to start fixing things like this now.

That changed the meaning of the entire exercise. The eventual customer wasn't buying an experimental prototype. They were buying a six-figure luxury product and quite reasonably expecting the software to reflect the badge on the front.

So how does an organization end up in a situation like that?

The tempting answer is incompetence. I've learned to look elsewhere first.

Earlier in my career, whenever I encountered a bizarre piece of code, a strange process or a management decision that appeared to make absolutely no sense, I tended to assume there was something I didn't understand. Surely somebody had thought this through. Maybe there was a constraint I didn't know about, or someone older and wiser knew something I didn't.

Experience has taught me almost the opposite. Sometimes there is a very good reason. Surprisingly often, nobody really thought about it at all.

There is also another category, and it is more interesting: decisions that look irrational from the company's perspective but make perfect sense from the perspective of the individual making them.

Think again about that automotive supplier. A serious stability problem has been discovered weeks before delivery. From the company's long-term perspective, knowing this is valuable. But consider the person responsible for the milestone. Delaying the delivery creates pain immediately: the project turns red, management gets involved, contractual penalties may apply, targets are missed and perhaps a bonus disappears.

Delivering creates a very different risk profile. Perhaps the software passes the requirements that were actually specified. The milestone is achieved, acceptance happens and the invoice gets paid. The OEM may discover the deeper problems later. Maybe.

One option creates personal pain now and with certainty. The other creates corporate pain later and possibly.

That asymmetry explains more than we might like to admit.

Back when our world was mobile phones rather than cars, software stability was commonly measured using MTBF, Mean Time Between Failures. By today's standards it was a fairly crude stability metric, but its purpose was sensible. A product might, for example, need to demonstrate 120 hours of MTBF before receiving a sales-go.

There were very good reasons for having hard gates. Physical products have physical schedules. Factory capacity is reserved, components are moving through supply chains, operators and retailers have launch plans, and marketing campaigns are already committed. Christmas, inconveniently, has never shown much willingness to move because your software isn't ready. So schedules mattered. Managers also had targets, and bonuses.

Suppose the required MTBF is 120 hours and the product isn't achieving it. The measurement has just done exactly what it was designed to do: it has produced uncomfortable information about the product.

The natural question is why it is failing. But we occasionally encountered another line of reasoning. Which test cases are causing the failures? Fair enough. And then: what would the MTBF be if we removed those test cases?

An impressively efficient way to improve software quality, provided that by software quality you mean the number in the management presentation. The actual software remains inconveniently
unchanged.

There was another proposal from the same era that deserves recognition for its mathematical elegance. Why wait 120 hours for an aging test when you could run three devices for 40 hours each? After all, 40 + 40 + 40 = 120.

The mathematics was sound. The understanding of software aging somewhat less so. The whole point was to observe what happens to one system as it ages. Three forty-year-olds do not make one 120-year-old, and neither do three phones.

It is easy to laugh at examples like these, but misunderstanding MTBF isn't really the interesting part. The measurement existed to tell management whether the product was stable enough. Once that information stood between people and something they were rewarded for achieving, the measurement itself became negotiable.

Goodhart's law says that when a measure becomes a target, it ceases to be a good measure. Add personal incentives and things can get considerably more creative.

One of my earliest lessons in incentives was subtler. I worked in a platform testing team, where our responsibility covered, roughly speaking, much of what happened underneath the phone's user interface. The UI had its own organization and its own testers.

We tested real devices manually, so naturally we encountered UI bugs as well. Our personal performance incentives, however, rewarded us for valid platform defects we were first to report and that subsequently got fixed. Platform bugs counted. UI bugs didn't. Nobody instructed us not to report UI bugs. There was no policy telling us to ignore anything outside our organizational box. But time and attention are finite, reporting bugs takes effort, and there was always a wonderfully convenient thought available: the UI testers will probably find it.

People therefore concentrated on the problems the system rewarded them for finding.

You don't need walls to create silos. Sometimes a bonus formula will do.

In management language, this is local optimization: every individual can behave rationally according to the incentives around them while the system as a whole gets a worse result. But there is another consequence that interests me more. The incentive isn't merely changing what people do. It is changing which information enters the organization.

The UI bug still exists. Someone saw it. Yet from the organization's perspective, it may as well be invisible.

Nobody decided to hide it. Nobody had to.

The system simply made reporting something else more worthwhile.

The same mechanism becomes more expensive when the boundary isn't between two teams but between two companies. An OEM wants an excellent product. Its supplier wants to deliver what was agreed, on schedule and profitably. Those objectives overlap, but they are not identical.

OEMs are generally quite good at specifying functionality. The radio shall do this. Bluetooth shall do that. Navigation shall behave like this. Those requirements can be documented, converted into test cases and eventually marked green.

Long-term software quality is less cooperative. After 150 hours of realistic continuous use, how much memory growth is acceptable? How much performance degradation? How many application restarts? How much resource accumulation or jank? What exactly does the system shall remain responsive mean in numbers? Who measures it, and what happens when the limit is exceeded?

If the OEM cannot answer questions like these, it hasn't merely forgotten a few requirements. It has surrendered part of the control over the quality of its own product. Someone else now gets to decide what good enough means, and very often that someone is the supplier.

Suppliers, unsurprisingly, behave commercially. They deliver what was specified, perform the work included in the contract and optimize their own economics. Expecting a supplier to routinely spend significant additional engineering effort finding and fixing things that weren't required, weren't measured and weren't paid for simply because everyone cares deeply about the end customer is a charming idea. It isn't a quality strategy.

I wrote about this problem years ago when working with mobile original design manufacturers (ODMs), using an old metaphor: the fox guarding the henhouse. The OEM owns the brand and the long-term customer relationship. The ODM needs to deliver the specified product profitably. Finding additional problems costs money. Investigating them costs money. Fixing them costs money.

Some commercial arrangements make the incentive even more peculiar. The supplier delivers the software, problems emerge later, and the customer then pays that same supplier for the engineering required to fix them. You have now created the remarkable situation where poor quality can generate revenue for the organization responsible for producing quality.

That doesn't mean suppliers deliberately produce bad software. They don't need to. If preventing a problem costs the supplier money while repairing it later generates billable work, perhaps the customer's quality strategy shouldn't depend entirely on the supplier voluntarily choosing the less profitable option.

The problem doesn't require bad people. The incentives are sufficient.

Automotive has another phrase I have heard in various forms more times than I care to
remember: It's a Tier-1 issue. (For readers outside automotive, a Tier-1 supplier is one that supplies systems or components directly to the vehicle manufacturer.)

No. It's your car.

The customer doesn't know who supplied your infotainment software, middleware or Bluetooth stack, and they shouldn't have to. You can outsource engineering, implementation, testing and even entire software platforms. You cannot outsource accountability for the product carrying your badge.

If you neither define nor independently measure the quality you expect, the problem goes further. The supplier builds the software and then effectively tells you whether the software it built is good enough. At that point, it's a Tier-1 issue isn't really assigning responsibility. It is revealing who actually has control.

And again, this is partly an information problem. If your only view of quality comes through the party whose delivery, margin or acceptance depends on that view, you may be receiving entirely accurate answers to the wrong questions.

Not every incentive appears on a payslip or in a supplier contract. Ego, ownership and simple self-preservation can be just as powerful.

Engineers have an impressive capacity for what psychologists call psychological ownership. We become attached to architectures, tools and systems we helped create. Organizations develop much the same attachment to processes. Someone designed the current approach, fought for its budget, selected the supplier or convinced management that this was the right way forward. Teams form around it, responsibilities settle and people build careers around knowing how it works.

We call our creations our babies for a reason. And your own baby is always beautiful, even when the available evidence suggests otherwise.

Then somebody arrives with a substantially better alternative. To the company, that should be good news. To the person who built, bought or championed the existing solution, however, the message may sound rather different: the thing you chose wasn't actually that good.

Technical evaluations can become remarkably creative at this point. Yesterday's annoying limitation becomes today's deliberate architectural choice. A capability the current system doesn't have turns out, after careful consideration, not to be particularly important. Black begins to acquire some fascinating properties of white.

There are obviously perfectly rational reasons not to change. Switching costs money, migration creates risk, and a slightly better tool isn't necessarily worth replacing something that already works. If you need a rifle, somebody enthusiastically offering you a cannon isn't automatically doing you a favor.

I'm talking about something different: cases where the fit is real, the improvement substantial, and accepting the evidence would also require somebody to admit that an earlier decision was wrong.

That can be a considerably higher barrier than technology.

We once saw an especially clear version of this. An organization had spent three months trying to understand a persistent class of software failures. Two task forces had been working on the problem without reaching the root causes.

A manager decided to commission a pilot using a different approach.

The pilot found the root causes in 24 hours.

The manager who had commissioned the pilot was delighted with the result. For a moment.

Then came the harder question: How do I sell this internally?

The established approach hadn't been chosen by him. The decision had been made several organizational levels above. Showing that a different approach had succeeded where the established process had struggled wasn't therefore merely presenting a technical result. It could also be interpreted as telling a senior manager that the approach they had chosen was wrong.

The technical problem had been solved. The organizational problem had just begun.

There are several familiar mechanisms hiding in that situation: status quo bias, sunk costs, hierarchy and psychological ownership. But underneath them is something simpler. People don't particularly enjoy admitting mistakes, and organizations often make changing course considerably more personally risky than continuing down a path that isn't working.

The higher in the organization the original decision was made, the more expensive proving it wrong can become. Not technically expensive, but personally expensive.

This also explains something that initially seems counterintuitive. A failed experiment is often harmless. A successful one can be threatening. If it fails, the existing system survives unchanged. If it succeeds, uncomfortable questions appear: why weren't we doing this before? What else aren't we seeing? Does the process need to change? Does somebody's budget, ownership or supplier revenue change? Was the previous decision wrong?

An experiment can be financially free and still be organizationally very expensive.

There is a well-known organizational tendency for unpleasant information to travel upward less enthusiastically than good news. One related concept in organizational psychology is the MUM effect: the tendency to avoid or soften messages we expect the recipient will find unpleasant. Corporate life has a less academic expression: don't shoot the messenger.

But what interests me isn't merely whether people are willing to speak up. It is what the system makes that information worth to them personally.

A serious problem discovered early may save a company millions. Yet the same discovery may turn someone's project red, threaten a delivery target, create months of additional work or challenge a decision made by someone several levels above them. A new technical approach may be excellent for the company while threatening the budget, supplier relationship or professional identity built around the old one. The information has positive value to the organization and negative value to the person carrying
it.

Corporate benefit later. Personal pain now.

Once you start looking for that asymmetry, a surprising amount of seemingly irrational corporate behavior becomes easier to understand. The company benefits from discovering a serious quality problem before launch; the manager whose milestone turns red may not. The organization benefits when a tester reports every relevant defect; the tester rewarded for one category rationally concentrates on that category. The OEM benefits from preventing long-term quality problems; a supplier paid to repair them may have different economics. The company benefits from replacing an obsolete process; the person who designed, bought or owns that process may have a more complicated relationship with the idea.

And organizations certainly benefit from learning that an important decision was wrong.

Finding the volunteer who wants to deliver that presentation can be harder.

None of this requires villains, conspiracies or even particularly bad management. That's what makes it interesting. Rational people responding rationally to local incentives can collectively produce an outcome that is completely irrational for the organization.

When I was younger, a bad software product naturally made me look for a technical explanation. What's wrong with the code? Why did this crash? Who introduced the leak? Why didn't testing find it?

Those are still useful questions. They just aren't always the most interesting ones.

Sometimes the code is bad because the architecture is bad. Sometimes the testing is inadequate because the process is bad. And sometimes the technical organization is behaving exactly as you should expect given the incentives surrounding it.

These days, when I see a decision that appears completely irrational, I find myself asking a different question:

What is the rational choice for the person making the decision?

And when important information seems to have been ignored, diluted or somehow lost on its journey through an organization, there is another question worth asking:

What happens to the person who brings the bad news?

Not what the corporate values say should happen. Not what the process diagram says should happen. What actually happens to their target, project, budget, authority, supplier relationship, reputation or comfortable status quo?

I don't think there is a neat five-step management framework hiding here, and I'm not sure I'd trust anyone who claims there is. Complex organizations, commercial relationships and human beings rarely reduce that gracefully into a PowerPoint slide.

But there is value in looking beyond the technical explanation.

When a software decision makes no sense, the code may not be the most interesting place to look. Neither may the process. Look at what happens to the people involved when inconvenient information arrives. Who benefits if it is acted upon? Who loses? Whose previous decision does it challenge? And who would have a considerably easier day if the information simply went away?

You may find that the organization isn't behaving irrationally at all.

The KPI can be green. The acceptance tests can pass. The milestone can be achieved. The bonus can be paid. The supplier can meet the contract.

And the customer can still receive terrible software. Not despite the system. Sometimes because of it.

Bad incentives don't only change what people do. They change what the organization gets to know.