Have we all been duped by the Myers-Briggs test?

[93JC]

Have we all been duped by the Myers-Briggs test? - Fortune

Have we all been duped by the Myers-Briggs test?
by Roman Krznaric
MAY 15, 2013


When Frank Parsons opened the world’s first career guidance center in Boston in 1908, he began by asking prospective clients 116 penetrating questions about their ambitions, strengths, and weaknesses (and how often they bathed). But then he did something more unusual: He measured their skulls.

Parsons was a committed believer in phrenology. If you had a large forehead, he might recommend you become a lawyer or engineer. But if your skull was more developed behind the ears, you were of the “animal type” and best suited to manual work.

Career advice has, thankfully, come a long way since then. But now, instead of measuring the outside of people’s heads, it has become common to measure the inside using psychometric tests. Personality testing has grown into a major industry and is standard procedure in leadership and management courses, as part of job-interview processes, and, increasingly, in career counselling. But should we really trust such tests to deliver scientific, objective truth?

I have some bad news for you: Even the most sophisticated tests have considerable flaws. Take the Myers-Briggs Type Indicator (MBTI), the world’s most popular psychometric test, which is based on Jung’s theory of personality types. Over two million are administered every year. The MBTI places you in one of 16 personality types, based on dichotomous categories such as whether you are an introvert or an extrovert, or have a disposition towards being logical or emotional (what it calls “thinking” and ”feeling”).

The interesting — and somewhat alarming — fact about the MBTI is that, despite its popularity, it has been subject to sustained criticism by professional psychologists for over three decades. One problem is that it displays what statisticians call low “test-retest reliability.” So if you retake the test after only a five-week gap, there’s around a 50% chance that you will fall into a different personality category compared to the first time you took the test.

...

One other thing, and this matters especially for anybody who thinks personality tests can guide them to a perfect career. According to official Myers-Briggs documents published by its exclusive European distributor, the test can “give you an insight into what kinds of work you might enjoy and be successful doing.” So if you are, like me, classified as INTJ (your dominant traits are being introverted, intuitive, and having a preference for thinking and judging), the best-fit occupations include management consultant, IT professional, and engineer.

Would a change to one of these careers make me more fulfilled? Unlikely, according to psychologist David Pittenger, because there is “no evidence to show a positive relation between MBTI type and success within an occupation … nor is there any data to suggest that specific types are more satisfied within specific occupations than are other types.” Pittenger advises “extreme caution in [the MBTI test’s] application as a counselling tool.” Then why is the MBTI so popular? Its success, he argues, is primarily due to “the beguiling nature of the horoscope-like summaries of personality and steady marketing.”

When I cite the avalanche of critical studies to career counsellors, coaches, and trainers who administer Myers-Briggs tests, they often point out that the test is not designed to match people to ideal careers. Yet many of them ignore the evidence and keep on handing them out, typically because they are still believers in it as a guide to personality types, but sometimes — I suspect — because it gives their advice a veneer of legitimacy.

(Bolded for my emphasis.)


I happen to agree with most of this. Your thoughts?

 

[EJCC]

Some people don't fit neatly into MBTI boxes. It's the nature of boxes that not everyone fits into them. However I'd suspect that the test-retest thing has more to do with the quality of the questionnaire, than it has to do with the test itself.

I totally agree with the point about career advice. It's amazing to me that the test is terrible at predicting one of the primary things it was designed to predict. Which is why people keep writing articles like this. My personal stance is that it's a marketing issue and not an issue with the MBTI -- made a thread about that a while ago.

 

[reckful]

That sounds like a pretty typical (straw-manny and poorly-informed) MBTI "debunking."

For a long discussion of the shortcomings of a more recent one, see this post.

 

[Seymour]

The interesting — and somewhat alarming — fact about the MBTI is that, despite its popularity, it has been subject to sustained criticism by professional psychologists for over three decades. One problem is that it displays what statisticians call low “test-retest reliability.” So if you retake the test after only a five-week gap, there’s around a 50% chance that you will fall into a different personality category compared to the first time you took the test.

Again, if you look at preferences as traits (and not dichotomies), the MBTI's rest-retest reliability is right up there with the Big Five instruments. So, yes, full types have low test-retest validity, but that's partially because many people's preferences fall toward the middle of the scale in practice. In the middling preference range, even a minor variation in MBTI preference scores can flip a type letter.

Still, I agree with the article (and EJCC) that career counseling is not generally a good use for the MBTI. Still, there are studies that indicate that job satisfaction in some particular careers are correlated with type (which contradicts the quotation in the OP), but there are better ways of predicting job satisfaction. There also don't appear to be studies for every career, so it's not like one can say matching one's career to MBTI type is always good.

Meanwhile, if we look at the Big Five studies on the topic of job satisfaction:

This study reports results of a meta-analysis linking traits from the 5-factor model of personality to overall job satisfaction. Using the model as an organizing framework, 334 correlations from 163 independent samples were classified according to the model. The estimated true score correlations with job satisfaction were -.29 for Neuroticism, .25 for Extraversion, .02 for Openness to Experience, .17 for Agreeableness, and .26 for Conscientiousness. Results further indicated that only the relations of Neuroticism and Extraversion with job satisfaction generalized across studies. As a set, the Big Five traits had a multiple correlation of .41 with job satisfaction, indicating support for the validity of the dispositional source of job satisfaction when traits are organized according to the 5-factor model

It does make sense across careers that extraversion is generally related to job satisfaction (extraverts are happier in general) and neuroticism is related to low career satisfaction. Looking at the sometimes correlations, one might wonder if the agreeableness and conscientiousness correlations tend to be more career specific, and that there might be other career-specific correlations for some career choices (and it appears this is the case for some careers). One can easily see how the corresponding MBTI preferences (excluding Neuroticism) might hold similar relationships.

In any case, the OP quotation reads like an over-generalizing hit piece, rather than something measured and thoughtful. The MBTI is definitely oversold in some cases and contexts, but claiming that it has no merit or scientific validity is underselling it.

 

[93JC]

The MBTI is definitely oversold in some cases and contexts, but claiming that it has no merit or scientific validity is underselling it.

What scientific validity does a Myers-Briggs test have?

 

[Seymour]

What scientific validity does a Myers-Briggs test have?

So, there are a number of ways think about how scientific an instrument is. One is test/retest validity. Another is construct validity, which is about whether an instrument measures what it claims to measure, and whether there are useful predictions (or at least correlations) which are validated empirically. Things like the MBTI (and the Big Five) are mostly about descriptive categorizations and have many useful correlations, so they are not scientific in the same way a mathematical law would be.

In truth, there are a number of studies which do show all sorts of interesting correlations with type. The pool of research is smaller than for the Big Five (and more often published by less neutral sources), but still it seems clear the preferences do correlate will all kinds of interesting things. When you add to this the correlations between preference pairs and four out of the five Big Five traits (the academic gold standard for personality), it's hard to argue that the MBTI has no validity and isn't measuring something real.

Of course, there are some valid criticisms of the MBTI. One issue is that the S/N and J/P scales of the MBTI are correlated, such that NP and SJ tend to pair, which shouldn't be true if each axis is actually independent. There are also indications that the S/N and T/F scales are a bit muddled (see the wikipedia entry for details). Type dynamics are a problem, too, since they don't seem to be measurable empirically, despite many efforts to do so.

Still, as reckful has pointed out, the creators of the Big Five have a lot of respect for the MBTI, and the MBTI appears to at least partially represent the fact that preferences (at least for dyads and triads) can be additive. The Big Five is oddly flavorless, in that each trait scale stands on it's own and is mostly just represented as a pool of adjectives (which makes sense, given the statistical origins of the Big Five). Given that additive nature of preferences (or traits, in Big Five terms), much better descriptions could be written about the effect of the combination of traits. The MBTI "functions" are examples of such descriptions, in that they effectively describe dyads or triads of preferences (where Te might be T+J, or E+T, or E+T+J).

Plus, the MBTI uses more neutral language than the Big Five, so is more publicly consumable. Being a Perceiver sounds not so bad, but being "low in conscientiousness" does not sound like someone one would like displayed on one's cube. Generally, the Big Five shows value bias towards high extraversion, agreeableness, conscientiousness, openness and low neuroticism.

So, while the MBTI is not perfect, it does appear to describe something real that has interesting real world correlations (at appears to be getting at some of the same stuff as the Big Five). The test/retest validity seems good, and the construct validity (while not perfect) isn't bad. It could probably be improved by removing type dynamics, increasing its alignment with the Big Five, representing the strength of preference, and expanding descriptions of preference pairs and triads (as Reynierse has suggested). But describing it as a scam or vastly less scientific than the Big Five is being unfair to the reality of the situation.

 

[93JC]

Oh I would not describe "the Big Five" as more scientific than Myers-Briggs.

 

[Seymour]

Oh I would not describe "the Big Five" as more scientific than Myers-Briggs.

In that case, maybe they are both on astrology-level footing for you. ;-)

Still, it's hard to argue that the Big Five doesn't correlate with tons of real-world stuff, allowing one to make non-trivial predictions (in the aggregate) about all kinds of things. It's also clear that situational effects often outweigh personality, so behavior is often determined by the extrinsic rather than the intrinsic. So your mileage may vary.

 

[skylights]

It seems to me that the baseline perspective of every person to any sort of tool in their life should be one of pragmatic skepticism. Seymour made an excellent summary of the usefulness and issues that pertain to the indicator, but I am still sort of personally floored when people don't take the MBTI (or anything in their life, really) with a dash of salt.

When a counselor uses a tool like this, they can glean information with an awareness of its limits. I don't think it's used because it gives a veneer of validity as much as it's used because it begins to crack away at discovering a person's preferences. The task a career counselor, coach, or trainer is presented with is that of matching a person to a role, and that is an extremely difficult prospect when they have never met the person before, and when the person themselves might not know what they want or like. This indicator is one method of creating a picture of who that person is and what they like and what type of environment would be well-suited to them. Hopefully, the tool, its inherent limits, and the results are all discussed by the professional and the client, to get a realistic picture of whether it will be helpful in that specific case or not. In the meanwhile, there are people working on developing better, more accurate, and more valid sorts of indicators. But, for right now, this is one tool that seems to have withstood some time for whatever reason.

What I have noticed in my own life is that these sorts of tests - MTBI, Big Five, Enneagram - tend to be roped into the workplace for an insufficient amount of time, such that essentially the professional can only administer the test and hand out a page of results. My mom, for example, took an Enneagram test at her workplace recently. They only had two hour-long sessions, the first in which they took the test and the second in which they received the results. There were approximately 10 people in her group. Divide that into an hour and you get approximately six minutes of personal attention each, and that is not accounting for an introduction or conclusion. She got a type that she felt like was okay, but it didn't fit her incredibly well, and she had no idea what to do with the results. We went over it together later, we found that her second most scored type fit her better, and we talked about some of the self-reflective and practical applications.

Without that sort of analysis and discussion at the end, I think typology is essentially useless. In my opinion, it's just shorthand for beginning to look at a person's strengths, weaknesses, preferences, and issues. And then from there, you can begin looking at how a person might fair in terms of performing and liking a certain professional role. But if you don't go further than the results, then yes, you are going to be duped, because the test isn't about the test, but about what the test points to and how that information can be useful in your life.

So, my feelings are that both the professional and the client should go into testing with an awareness of what the tool is and what its limitations are, and after the results have been presented, there should be a thorough unpacking of how the client feels about the results. It is the responsibility of the professional to make those limitations clear to the client, but I also think that it is the responsibility of every individual to be aware of what sort of information and sort of opinions are entering into their lives and to be aware of the stance that they are taking towards that information and how much they are going to let that information influence their life or not.

 

[Bush]

Oh I would not describe "the Big Five" as more scientific than Myers-Briggs.

Do you mean that the Big Five isn't scientific in that it's bogus and/or useless in practice, or that it wasn't developed with scientific methods in mind?

The latter's patently not true; but for the former, yeah, it could be bogus. Its bogusness is contingent upon how it's used. e.g.

It seems to me that the baseline perspective of every person to any sort of tool in their life should be one of pragmatic skepticism. Seymour made an excellent summary of the usefulness and issues that pertain to the indicator, but I am still sort of personally floored when people don't take the MBTI (or anything in their life, really) with a dash of salt.

When a counselor uses a tool like this, they can glean information with an awareness of its limits. I don't think it's used because it gives a veneer of validity as much as it's used because it begins to crack away at discovering a person's preferences.

This.

That is to say, if it's used for anything else beyond the scope for which it's been validated, it really is going to be bogus and unscientific in practice.
__

Plus, the MBTI uses more neutral language than the Big Five, so is more publicly consumable. Being a Perceiver sounds not so bad, but being "low in conscientiousness" does not sound like someone one would like displayed on one's cube. Generally, the Big Five shows value bias towards high extraversion, agreeableness, conscientiousness, openness and low neuroticism.

I see this as one leg up that the MBTI has over the Big Five. In fact, I believe that the MBTI ought to be thought of as less scientific, though with that less scientific scope in mind. You know, use it for softer and lighter stuff: 'feel-good,' team-building, career-guiding.

We're duped by the MBTI if we see it and use it for anything more.

Otherwise, if the goal is to make MBTI more scientific in representation, and if that goal coincides with "Oh, hey; it measures a lot of stuff that the Big Five does!", then why not cut out the middleman and simply use the Big Five?

 

[reckful]

Otherwise, if the goal is to make MBTI more scientific in representation, and if that goal coincides with "Oh, hey; it measures a lot of stuff that the Big Five does!", then why not cut out the middleman and simply use the Big Five?

Perhaps because, as McCrae and Costa (the leading Big Five psychologists) have acknowledged, each typology may have things to learn from the other.

 

[skylights]

This.

That is to say, if it's used for anything else beyond the scope for which it's been validated, it really is going to be bogus and unscientific in practice.

I agree. Though my fear of harmful usage of it is more directed towards the business world than the scientific world. Businesses incorporating the MBTI - or any other personality inventory - to try to sort or make decisions about their employees have more of a dangerous prospect on their hands in terms of using this to box people in and limit their opportunities, I think. I have already seen touches of personality-role matching questions at my previous job and it just made me shudder to think how completely inaccurate it could end up being, because I know no one who knows psychology from a monkey's ass is working in that hiring office.

 

[Bush]

Perhaps because, as McCrae and Costa (the leading Big Five psychologists) have acknowledged, each typology may have things to learn from the other.

There's a lot of discussion here on what the MBTI can learn from other systems, but, as you mention, there's discussion about the flipside too.

On what the MBTI can teach other personality systems, they state that its neutral, colloquial language makes it capable of "teach individuals the nature and value of individual differences"; and that existing "extensive empirical literature on the MBTI" in terms of career choices could be assimilated into our collective understanding (p37).

That is, what MBTI has to teach us is (a) the non-scientific 'understand one another' stuff, and (b) the results of the studies that have already been conducted with the MBTI. I believe that MBTI has a good amount to offer for (a), and that 'non-scientific' doesn't mean 'absolute garbage' (it's useful stuff), but for (b)? Its existing studies can be incorporated to great benefit, sure; but I believe that future studies could be conducted with an instrument that's better suited for the task.

That instrument may or may not be the Big Five. HEXACO seems promising--like MBTI, it hits on most of the Big Five; but it adds another dimension, which may give a better explanation of personality--though I don't know too much about it.