Leadership and the constant danger of Overconfidence

A Shelley Gare article in the SMH Good Weekend (25 May 2019) throws doubt on the success of the decades of academic and consultancy effort intended to improve leadership skills in business.   Titled “How phonies and self-promoters came to rule the world” Gare writes that the world has become so obsessed with money and so susceptible to ‘charisma, over-confidence and surface gloss’ that it has invited ‘sham, spin, trickery and twaddle’ to become business norms.  This deterioration in business ethics would have to be regarded as a failure of business leadership.

The Gare article instances critical organizations neglecting their important mission through an obsession with branding exercises and “management claptrap”.   It quotes corporate insiders and recruiters as decrying the rise of “the leadership cult in business” and the consequent tendency for those at higher levels of the corporate ladder to spend their time promoting themselves rather than their organization.   Narcissism is further fed by directing corporate budgets into internal corporate communication teams that are focused on building the reputation of the CEO, and have led to instances of high profile failures where leaders who performed brilliantly when scripted and rehearsed by their corporate communications people proved incompetent when things went off track. Rod Sims, head of the Australian Competition and Consumer Commission, has pointed to the problem of arrogance in much of corporate Australia.   He says “they are in a privileged position and can do as they like in an unfettered way.”  Sim’s view of corporate arrogance was vindicated by the litany of leadership failures uncovered in the 2018 Banking Royal Commission.    But leadership incompetence caused by overconfidence, arrogance and hubris are not things confined to Australia.  The Boeing 737 Max 8 scandal is a recent example of similar incompetence at the highest level of global business.

The research of Professor Tomas Chamorro-Premuzic, a globally recognised authority on management performance and careers, (refer his articles in Harvard Business Review, TED talks on YouTube, and books Confidence (2013) and Why Do So Many Incompetent Men Become Leaders? (2019)), is informative about these issues.   Chamorro-Premuzic argues that, around the world, leadership success rates are woeful.  As evidence, he cites a large global study of 14,000 HR professionals that gave positive ratings to only 26% of their current leaders.  And his books point to widespread leadership problems that result from the human error of mistaking confidence for competence.

Chamorro-Premuzic makes this simple point:   Referring to someone as overconfident and self-absorbed would normally not be thought complimentary.   However, observing a person with these traits will more often than not prompt the thought that the person is charismatic and a potential leader.   Consequently, Chamorro-Premuzic says, people tend to equate leadership with the very behaviours (overconfidence and self-absorption) that lead to poor leadership, and, in doing so, often neglect the crucial question as to whether the person knows their stuff (are competent). People are led into this mistake because confidence is so highly regarded in business and it is not easy to distinguish between confidence and overconfidence.  Yet people do not normally make this mistake in choosing a dentist, doctor, or airline pilot.

We have learnt to observe and accept over-confidence in our political figures.   A politician lacking confidence would probably have difficulty in getting elected and there is sufficient ambiguity in politics for over-confident claims to be tolerated (there is always someone else to blame).   However in business, overconfidence that manifests in arrogant and risky behaviour, and thus leads to potential leadership and business failure, would hopefully be identified and corrected.  However, we’ve all met someone in our work or social life that was obviously incompetent in a particular subject but confidently insisted that they knew everything about it.   And we may have observed such people being promoted to positions beyond their competence, with disastrous results.  This is the type of dysfunction that Chamorro-Premuzic reports.

Research on the confidence-competence question was initiated 20 years ago when two psychologists, Dunning and Kruger, conducted a simple experiment that established that most people overrate their own capabilities because they have no way of measuring their own level of ability and so tend to exaggerate their self-assessments. Put simply, most people are not as smart as they think they are.   Psychologists call this the cognitive bias of “illusory superiority” – the feeling that we know more than those around us.  Reported surveys showing 90% of people classing themselves as “above average” drivers is an example of this effect.

In Dunning and Kruger’s experiment college students were given several categories of tests that measured their abilities in writing, reasoning, and thinking. After the tests were scored the students were asked to estimate their personal scores, the difference between the estimate and actual score becoming a measure of the student’s confidence.  The intriguing result, now termed the Dunning-Kruger Effect, was that the less competent students overestimated their results while those students who performed better at the tests underestimated their result.  In other words, the students of lower competence displayed the higher levels of confidence while the students of higher competence displayed the lower levels of confidence.  This result can be explained.

Lesser skilled people are less aware of the full extent of the particular subject and so have less appreciation of how much they don’t know.   With their narrow perspective of the subject, the less skilled tend to believe that they know everything necessary to know, and so tend to be overconfident about it.

In contrast, highly skilled people are more aware of the vastness of the subject they are working on, and so can better appreciate what they don’t know about it.   Consequently, with their more complete perspective, they usually underestimate their knowledge and competence in the subject, and so display less confidence in their decisions.   The ancient Chinese philosopher Confucius’s counterintuitive thought that, ‘true wisdom is knowing what you don’t know” confuses people to this day.

The differences between the less and the higher skilled in what they don’t know is also reflected in their different approaches to learning and improving. In contrast to higher skilled people, the lower skilled, believe that they know everything they need to know, and so are less responsive to feedback that suggests that they need to improve. The ancient Greek philosopher Socrates’s thought that “smart people learn …. stupid people already have all the answers” also continues to confuse people.

In working through the implications of Dunning-Kruger what must be kept in mind is that all research is adamant that long-term success in any activity, be it education, business, sport, or other, relies on competence, the ability to accomplish the task to the required level of success.

But that does not deny that confidence, people’s feelings about their ability to accomplish a particular task, is not important, but its effects on performance differ.  People with high confidence believe themselves fully capable of succeeding; people with low confidence constantly question their own ability to succeed and so are more responsive to calls for improvement.  Chamorro-Premuzic claims to have examined the career data of over a million individuals across different fields of competence (business, education, arts, sports, and criminal activity).  His conclusion:  Career success always depends on the same three factors – demonstrating competence, working hard, and being liked.  Despite the business bias towards higher confidence it is not always advantageous.  There are two reasons for that.   First, as Dunning-Kruger demonstrated, lower confidence people are more likely competent and more likely to want to improve, both things being necessary for career success.  Second, highly confident (and especially overconfident) people are often viewed as arrogant, and that puts them at a career disadvantage to the more humble, who are more readily liked and so better able to elicit the cooperation of others.

We have to understand confidence as having two faces – an external face (display) and an internal face (delusion). The benefits of confidence arise through its display to others.  Confident people are seen to be more extroverted and socially skilled, and these things are advantageous to effective leadership in obtaining the commitment of employees, customers and stakeholders.  So why would people not be encouraged to display confidence?

However, this bias towards confidence becomes counterproductive when people come to regard displays of confidence as an indicator of competence, for while the Dunning-Kroger experiment and other research have proven this bias to be false, in most situations confidence is more easily discernible than is competence, so that in recruitment interviews, for example, confidence can be used to mask incompetence and lead to more competent individuals being passed over in favour of the more confident.

Dunning-Kruger also points out the downside of confidence – its basis in delusion. The cognitive bias of “illusionary superiority” means that most people are chronically overconfident.  Chamorro-Premuzic recommends that confidence should always be treated as a delusion – one that people use to mask their lack of competence.  The important effect of such delusion is in biasing a person’s decision making.  Deluding yourself into thinking you’re better than everyone else is not a great strategy for success, whatever the activity.  It increases your propensity to take excessive risks, and that can lead to tragedy for yourself and your employer.  It can lead to you making claims of confidence that you cannot deliver, and when confident claims are revealed as false it backfires on you.   Credibility, trust and reputation can be destroyed.  And as a leader your overconfidence can permeate your organization and create a sense of invincibility or hubris that can bring your organization undone.

The questions for any leader are: How can you benefit from the advantages of displaying confidence but avoid the dangers posed by overconfidence?  How can you overcome the self-delusion that underpins the Dunning-Kruger effect?  An answer lies is in controlling and regulating your self-confidence.  The following are suggestions for doing that:

1. Let go of your ego:   It is necessary that you are fully self-aware of the strengths and weaknesses that define your professional attributes.   While the Dunning-Kruger effect might say that you are probably not as smart as you think you are, Chamorro-Premuzic’s research suggests that becoming more humble will be advantageous to your career.
2. Listen to humbling evidence:   Paying attention to evidence of your shortcomings might not boost your ego but it will go a long way towards improving your judgement and avoiding costly mistakes.   The trap is that people, when finding they are not ranked as highly as they thought, typically disregard the clear evidence in front of them.  To escape this trap, you must be prepared to rely on a trusted advisor to tell you whether you are taking into account the unfavourable evidence right in front of your face.
3. Get people to show you what you don’t know:   As the ancient philosophers advised us, people’s wisdom is more limited by what they don’t know than what they think they know.   In the Dunning-Kruger research students’ delusion of confidence were reduced by tutoring on the skills they lacked, so demonstrating the value of having someone ahead of you showing you what you have yet to learn. That someone can be a trusted advisor or mentor, or expert people in your organization, academia or industry that you might consult.
4. Replace introspection with the “outside” view:   A way of protecting yourself against “delusions of success” is by taking an outside view of things (refer Lavallo & Kahneman, HBR, July 2003).  It is intuitive to evaluate your performance through your own thinking processes.  But this “inside view” is inevitably biased.  The “outside view” counters the effect of cognitive bias.  To take a simple example, if you ask a builder to estimate how long to build a house from the details of your plan he might say 3 months (an inside view). At the same time you telephone the local council to find that the average house takes 9 months to build in that locality (an outside view).   What estimate is most realistic?  The fact is that in most planning contexts the outside view delivers more realistic (or less unrealistic ) estimates than the inside view.   Use the outside view by benchmarking yourself and your career progress against relevant external people and information. It is worth remembering that your career progress relies less on what you think of yourself (inside view) than what influential others think of you, that is your reputation (outside view).
5. Ask yourself if you are really in control:   While we always have some degree of control, we must accept that a large proportion of our success results from factors beyond our control.   It is when we have little control over outcomes that we are at risk of over-estimating our degree of control and entering into competitive domains (markets, careers) in which we assume we have control.   However, when our rivals do the same thing we face more competitors and the outcomes are more dependent on the performance of these competitors and less on ourselves.   So overestimating our own control of outcomes can lead us into domains where we have less chance of success.  For example, in a rising share market there are times when all shares rise.   Being successful in this context is not a sign that we should change jobs.   A realistic assessment of how much control we really have will improve the efficacy of our decisions.

Dr John Waters and Mark Rosenberg